Music Processing, from Stage1 to Final Edition ===================================================== Created: 02/23/06 Revised: 04/28/09 The propose of this documention is to describe all of the steps in the process of going from a set of primitive stage1 files to a final edition of music. What is laid out here is a continuous work flow process; but the reality is that it is almost never followed in its entirety. Almost all works, in fact, spend a good deal of time in a partial state of completion; many will never reach the final state of a musical edition. The real purpose of this documentation, then, is to gather in one place all of the instructions for the various stages of data entry and processing. Note: Unless otherwise stated, all programs referred to in this document are stored in the directory j:/release/progs. On our Linux systems, this directory is included in the searchable list directories for zbex progs, which means they can be run directly by name without typing zz first. I. Initial Data Entry ────────────────────── (creating basic Stage2 files) The final result of the Intial Data Entry process for a musical movement (or piece) is a basic set of stage2 files. At the moment, there are three ways to get there: 1. Create a set of Stage1 files, and then convert these to stage2. Stage1 files are single track only, and do not support such things as time changes or variable length measures. Stage1 files are the output of the program input.z (with midi enabled). Note: input.z is a combination of the old a program for midi data entry, the old b program for processing midi data, and the old look2 program for displaying the result. At the moment (04/28/09) this program runs only in the old dmuse environment on a DOS machine. Hardware is setup for this at the Middlefield office. Once created, Stage1 files may be converted in a generic way to the Stage2 format using the program begin.z. Note: I recently did this with some stage1 Bach organ music, and the process worked perfectly. 2. Create a set of Stage2 files, using the commercial program Sharp-eye. This process is currently used by Don and Ed at CCARH. Basically, a peice (movement) is scanned by Sharp-eye and the Music-XML output is sent to the program Finale. Finale has a plug-in which converts this to Stage2 files. The scanning process involves some minimal editing. The process works only with printed music. The final Stage2 output requires some editing to bring the files into conformance with accepted practice. 3. Create Stage2 files by manually entering them in a DMUSE window. This process is slow, and is not recommended for large projects. For small projects, it does have the advantage of flexiblity. I(b). Setting up the file organization ────────────────────────────────────────── This process is implied in creation process, but it is essential that it be done at this point and that it be done right. All Musedata files are organized according to the following system: work/ --> use any name you like stage2/ name must be stage2 mvt1/ --> use any name, but numbers are the default 01 --> use any name, but numbers are the default. 02 The group structure determines the use and ... meaning of a file. See s2-specs.ten in j:/release for details on group structures nvt2/ ... outputs/ name must be outputs score/ --> The names at this level reflect the purpose and score18/ size of the music being compiled. Letter portion short/ of the name must match one of the group names. skore16/ The number portion is the notesize. Size 14 is parts/ the default and is not needed. parts21/ score/ (example) formats name must be formats i-files/ name must be i-files mvt1/ --> these names match the names in the Stage2 directory 01 --> these file names are created by autoset, and are 02 always numbers. These are linear i-files for 03 the various parts in the score. ... mvt2/ ... pages/ name must be pages mvt1/ these names match the names in the Stage2 directory 01 these file names are created by mskpage, and are 02 always numbers. These are page specific i-files. ... editions/ any name is possible, but editions is preferred name1/ can be anything, but should be descriptive, e.g. 1st-2004/ score/ any name, but descriptive. This is where the .mpg edition pages are stored. (some more examples of edition directories) cont21/ xcont21/ parts/ scform --> this is the "make" file for the edition. For no particular reason, we have standardized on this name. The scform file can actually be stored anywhere on the system, since all references to directories are by full path name. The scform file here is for building a score. The scform files for building parts are stored in the outputs/parts/pages/ partname/ directories score.cft CFT compression of files in /score mes.srh any name, but with srh extension. This is the compressed search file. postout.tmp/ this is a temporary directory used by the program pspage.z to store .ps postscript file. ednum this small file referenced by pspage.z to get the CCARH edition number and production date. score/ (example) 001.mpg These are the pages in a score. They should be numbered 002.mpg and should have the .mpg (music page) extension. parts/ (example) vln1 at the moment, there is no standardization on part names 001.mpg Notice that with the parts directory, there is an extra 002.mpg level of files. ... vln2 ... distrib/ name must be distrib kern/ must be kern: kern files for each movement (by name) midi1/ must be midi1: midi1 files for each movement (by name) midiP/ must be midip: midip files for each movement (by name) mused/ must be mused: MUSEDATA files for each movement (by name) score/ must be score: score files for each movement (by name) mvt1/ --> use any name, but numbers are the default mvt2/ ... mvtn/ doc Optional file containing history/current-state of data notes Optional file containing critical notes on data entry. Sorry about this long list. We are getting a bit ahead of ourselves here, but it is important to show at the outset how the data structure should be set up because all processes further down the line depend on this structure. I haven't shown the structure above work/ because this will depend on what piece is being entered. This description belongs to another discussion on the overall organization of CCARH data -- a very important subject, by the way -- and one which has not, in my opinion, been worked out in satisfactory detail. The problem, in a nut shell, is that we do not yet have a clear idea of what the public's view of our data should be -- although we have made good progress here. II. Supplemental data entry on Stage2 files ─────────────────────────────────────────────── This is actually an ongoing (and potentially never ending) process. The Stage2 file is fundamental data source for all other music files, whether sound, display, print, or analysis. But the Stage2 files that come out of Initial Data Entry often need a number of things added, including slurs, articulations, ornaments, transpositions, re-arrangement of stems and beams, clef changes, track numbers, print suggestions to improve the visual layout, sound suggestions for more accurate sound files. In addition, the group structure may need to be adjusted. Cue-size notes may be added to parts, etc., etc. If Supplemental data entry is in an early stage, i.e., many basic things need to be added, the program s2ed.z provides a graphical means for doing this, part by part. Otherwise, changes to Stage2 files must be done by hand. For Stage2 files representing more than one instrument, or for music on the grand staff, it may be necessary to merge two or more Stage2 files. There is a program called fmerge.z that purports to do this, but it is flawed in many respects. The directory franfro inside release/progs contains some programs and a process which is more careful about merging files. The more careful the program, the less work required to fix the result. There have been some recent changes way multiple tracks are represented in Stage2 files. (1) only use the backup command when a track reaches the end of a measure. If the track does not reach, use the irest to complete the track. This is necessary not only for score conversion, but also for making .srh files (2) When representing multiple instruments, do not use chords unless it is absolutely necessary. autoset.z will construct chords when the music isorhythmic and the stems point the same direction. (3) the global print suggestion for rest placement can save a lot of time and space; try to use it. III. Create initial output displays ─────────────────────────────────────── This is actually an interative process. The initial objective is to see the music in print and to use this to correct errors and to enhance the music. The final object is to get the stage2 files into the most advanced state possible, before moving on to the Edition stage. The basic sequence is autoset.z -> mskpage.z -> dskpage.z. The first object is to get these programs running without error. This will correct 99.9% of all format errors in the stage2 files. III(b). Determine the system layout ─────────────────────────────────────── The system (and page) layout is influenced by the spacing of the notes (compression factor), and by degree to which various empty staff lines are dropped from the system (this also depends on the system breaks. Our software does not have a "top down" way of arranging system breaks and page breaks. There is a print suggestion (on the measure record) for forcing a system break, but the overall effect on system breaks is still left to chance. The only way we have of deterining a system layout we like is to run the autoset.z -> mskpage.z -> dskpage.z sequence several times, with different compression values and possibly different line spacings (e.g., contained in the files in outputs/formats). This iterative process has, to date, always been successful in producing a satisfactory system layout. But it is essential that this process be completed to the user's satisfaction at this point in time, because all future editing will depend on a stable system layout. The system layout can be changed manually, of course, but this creates what we call an "orphan" result -- that is, a result that is no longer tied back to the original stage2 files. It is of great stategic importance that this connection be preserved, because we would like to preserve the ability to make small changes in the Stage2 files and then run the autoset.z -> mskpage.z -> dskpage.z sequence and other down stream processes again to get an "updated" Edition automatically. IV. Make adjustments to vertical and horizonal positions ────────────────────────────────────────────────────────── Vertical adjustments and minor horizonal adjustments are made with print suggestions in the Stage2 files. These can be made at any point in the process. Major horizonal adjustments to note spacings can be made using the xmskpage.z program, but because of bugs in this program, we recommend that adjustments are made only once, and then not changed. The program displays each system as it is typeset, and allows the user to move objects horizontally to improve the layout. The result of this editing is stored in the formats file for this movement. Thereafter, the basic typesetting sequence should be autoset.z -> xmskpage.z -> dskpage.z. Horizontal editing results will be preserved, as long as you don't try to make further changes. At some point, I hope to make this process more robust, so that further changes can be added. V. Make an Edition from the Page Files in OUTPUTS ───────────────────────────────────────────────────── There are several shortcomings with the page files in /outputs (1) Each movement is in a separate directory (2) Page bottoms are not uniform (3) Paging information is absent (4) The vertical arrangement of lines and systems is not optimal (5) Text underlay sometimes bumps into notes In earlier times, it used to take several weeks to cobble together an edition from the page files in /outputs/pages. Each page had to be separately edited and copied into an /editions directory. Now there is a faster way to do this work, and a means by which the process can be automated, so that if (or when) changes are made up-stream (e.g., in the Stage2 files), the edition can be recompiled without doing all of this work again. The key to this process is an instruction file, the format of which is documented in release/progs/edform.doc, and which for the time being we are calling scform. The file provides such information as: source lib: pages library providing the source for the edition destin lib: library into which output pages are to be stored blank page: generic message to put on any "blank page" page id header: location and font information for page id page id format: page id, for every page. Includes a page number top 7 "Z" records, required for .MPG file format The file then provides, on a page by page basis, the page number, the exact location of the source, and a set of (encoded) instructions on how each musical staff line "L" is to be adjusted up or down, and also how to adjust the vertical height of text underlay and a line of figured harmonies, if either exist and need to be moved. These instructions are optional; the page layout can be moved without any adjustment. But if the edition is to look "professional," the tops and bottoms of the musical page should be uniform across the edition. Simple instructions such as: VJ-1 = vertically justify proportionally to 2800 VJ-2 = vertically justify proportionally to 2800; with x2 space between systems VJ-3 = vertically justify proportionally to 2800; use only inter-system space can do this. More complicated instructions address individual lines. The scform file must be generated by hand, because the design of the page layout and the individual files (with their separate movement libraries) cannot be reliably constructed by computer. The moving instructions, however, can be generated by a program. This program is called vskpage.z, and is a variant of the page editing program eskpage.z, about which more will be said in a moment. vskpage.z is a full page graphics editor of a musical page. The only editing commands which work, however, are those which move the music staff lines and music systems up and down, and those which move text underlay and figured harmonies up and down. The output of vskpage.z is a line of instructions, displayed on the screen, which must be transferred to the screen in which the scform file is being constructed. This process may seem somewhat awkward, but it is flexible and provides the user complete control over what goes in the scform file. Once the scform file is built, the program that actually constructs the edition is called vspace.z. You can run vspave as many time as you like, without changing anything in the source pages. In this way, by adjusting the scform file and its moving structions, you get get the edition to look exactly the way you want. Just to summarize an important point here; using the scform file, the vskpage.z and vspace.z programs can build an edition from separate pages libraries and can adjust vertical heights to avoid clashes between musical lines. But they cannot change the system breaks or page breaks, and they cannot change the horizontal or vertical layouts on the staff lines themselves. These processes must be completed in steps II, III, and IV. However, it is possible to return to earlier steps, modify things, and run the vspace.z program again to get an updated version of the edition, page by page. In other words, the makeup of the edition is determined entirely by the content of the Stage2 files, the files in the /outputs/formats directory, and the scform file used by the vspace.z program. Why am I repeating all of this? Because there is one more step in the process. VI. Modify pages in the Edition ───────────────────────────────── Once an edition has been constructed, it is still possible to edit the individual pages to further improve the "presentation." The program that does this is the full screen graphical music editor eskpage.z. The input and output of this program is the same: a page-specific i-file. Note: It is also possible to edit the i-file manually in a normal window, but this requires detailed knowledge of the i-file format. A musical page can always be improved (sometimes just a little, sometimes a lot) using eskpage.z. The problem is that eskpage.z creates an "orphan" result; that is, the record of what has been done is contained only in the results, and not memorialized in the process (as in the case of the formats file or the scform file). If a change is made upstream, say in one of the Stage2 files, it cannot be made to propagate all the way to the final result. For this reason, we make two recommendations: (1) try to make all improvements before Step VI. Most position improvements can be accomplished with print suggestions in the stage2 files. Most horizonal spacing improvements on staff lines can be made with the smskpage.z program and stored in the formats file. Vertical arrangement of staff lines can be adjusted in the scform file. What else is there? (2) Run eskpage.z only on editions that are to be "set in stone," i.e., you don't expect to update them, but expect rather to make a new edition if further changes are warrented. Two examples of this would be (1) a typesetting of music to be produced in hardcopy, e.g., a Dover Edition; (2) a musical score that is being produced for a specific purpose (e.g., a performance); (3) semi-permanent scores and parts to be published on the web. VII. Postlude 1: Create a .CFT compression ────────────────────────────────────────────── Once an edition of .mpg pages is made, it is possible to make a single compressed file of the entire edition. The Dmuse environment has a utility to do this. VIII. Postlude 2: Create a .SRH search compression ──────────────────────────────────────────────────── I am currently working on a format for a compressed search file (extension .srh). The current program name is pagecomp.z. It operates on an entire edition. The program that does the searches is still work-in-progress. (Old location: R:\NEWMAR\EXPER\P08.Z) IX. Postlude 3: Create a set of SCORE parameter files ─────────────────────────────────────────────────────── The autoset.z -> mkspage.z -> dskpage.z sequence has been modified for the purpose of creating SCORE parameter files corresponding to the page-specific i-files in outputs/score. The new sequence of programs is autoscr.z -> paging.z -> scorecon.z. The group name is scrcon, so the output goes to outputs/scrcon. If you look at the i-files, you will see that they correspond exactly to the i-files created by the normal sequence, but with extra information appended to many of the file records. This information is used by scorecon.z to produce .pmx and .pmy score files in the directory /outputs/scrcon/pages. X. Postlude 4: Create a set of MIDI files ─────────────────────────────────────────── MIDI files can be created for each movement from the Stage2 data using the program midicon/mkmidi.z. The process requires some sound records and also a specification of channels and instruments, which are stored in the Stage2 library and format under the name name midicon. XI. Postlude 5: Create a set of TIFF files ──────────────────────────────────────────── The DMUSE environment provides a means of creating TIFF files on a page-by-page basis. These can then be converted to pdf files using commercial software. XII. Postlude 6: Create a set of .ps files ─────────────────────────────────────────── We now have the ability to create .ps files directly from page specific i-files. The program pspage.z is a variant of dskpage.z. Because these files relate directly to editions begin published on the web, I have devised a file organizational structure that deals with these files directly. The top of the structure is the directory dataout in c:/usr/local/apps. The structure looks as follows: dataout editions postscpt admin progs contents (contains more documentation) beethoven orch sym1 score parts cb ... sym2 ... chamber ... haydn mozart bach handel ... pmx-form (proposed only) cft-form (proposed only) midi (proposed only) humdrum (proposed only) ... The pspage.z program should be run from the edition library that contains the source data. An example might be: j:/musedata/classical/beethoven/bhl/orch/sym7/editions/beta-2008. This library must contain a temporary output library called postout.tmp, and a small file called ednum, which contains an assigned edition number for this conversion and a date. The program will ask for the library containing the .mpg pages you wish to convert. Examples might be score, or parts/vln1. The output .ps files go into postout.tmp, and you (the user) must then move them to the proper location in the dataout directory. At the moment, you (the user) are also responsible for clearing the postout.tmp library before each use. Otherwise, previous files that are not over-written will still be there (i.e., page data from other parts). I deal with this problem by renaming postout.tmp directory to the name I want in the dataout directory and then moving the whole thing there. I then re-allocate a new postout.tmp, which will by default be empty.