One of the major constraints of the development team of Koha is to develop a "multi-MARC" application.

But although all declinations of the MARC format have the same form (ISO2709), the meaning of each tag and it's sub-fields can be completely different from one dialect to another.

For example, in UNIMARC, the title is stored in 200$a.

In MARC21, it's in 245$a.

Whereas the 245$a of UNIMARC contains ... nothing (the field simply doesn't exist !)

Well, this means that it's impossible to administer all the differences in a simple way.

The Koha development team has thus chosen a method to circumvent this problem, without putting a strain on performance: everything is stored twice:

in "MARC" format (200$a which ignores the meaning of the field) and in a "decoded" form (title which ignores the MARC position).

A table for parametrization is set up at installation. This table contains all the labels for all MARC21 (or UNIMARC) fields and subfields, as well as the links between the two databases.

It's the application which takes care of storing all data twice, namely at the creation or modification of a record.

Even if this adds a lot to the complexity of the operations for record creation/modification, it simplifies the operations of search and reading just as much. Since in an ILS over 90% of access operations are for search and less than 10% are record creation/modification operations, the speed of the application is reinforced even more so.

For example, for the display of a search result of a query, the application uses the "title" directly. If the MARC format was used, the whole record would have to be searched in order to only extract it's title!

Note that Koha is capable of managing different frameworks. I recommend refining the first framework before creating subsequent ones.

At first, we are not looking at authority records.

Select the default framework and modify it's fields one after the other.

For each MARC tag, the following information has to be defined:

For each subfield, the following information has to be defined:

There are also 3 options to add constraints for the data input in subfields:

These elements can be neglected at first, they'll be treated later.

Now, go to the cataloging menu and add an empty record. Verify if you have indeed activated all the options you want to.

The links between the two databases MARC and non-MARC can be verified easily through

Koha >> Parameters >> Links Koha - MARC DB

This window lists all the connections of the fields between the non-MARC and the MARC fields/subfield.

The links can be modified in this interface, but beware: each modification is valid for ALL frameworks.

As soon as the script has finished running or if you have interrupted it, you can have a look at the catalogue.

Normally, the first result won't look great. Search for a frequently used term and check whether there are any item records (in the simple view as well as in the full view all items are regrouped in a special "group").

If the item records are missing go back to parametrization of Koha so as to make item records appear.

Verify also if both the "simple", i.e. non-MARC, and "complete", i.e. MARC, view contain data.

If subfields seem to be missing in the MARC view on display, two things are possible:

If data appears in the "complete" (MARC) record but not in the "simple" (non-MARC) record, this can be OK, or not.

Actually, the number of fields which can appear in the "non-MARC" part is limited. Only the following fields can be found there:

Alls these zones are NOT repeatable in the "non-MARC" part of the database, the part which appears in the display of the simple view. If one of these zones is repeated, only the first one will be displayed or in certain cases, the zones will be separated by the | character.

Three further zones are repeatable:

It's also possible to make some SQL queries for data verification.

The following query will display all the fields and subfields with their use. This query is quite helpful in detecting errors of the parametrization of the cataloguing framework:

SELECT tab, tagfield, tagsubfield, count( * ) AS tot FROM marc_subfield_structure 
LEFT JOIN marc_subfield_table ON marc_subfield_table.tag=marc_subfield_structure.tagfield 
AND marc_subfield_table.subfieldcode=marc_subfield_structure.tagsubfield 
GROUP BY tab, tag, subfieldcode

It's possible to iterate the imports of the catalogue as long as needed, i.e. until a satisfying result is obtained.

Note that the script bulkmarcimport.pl does not modify the bibliographic records.

A migration can by the way be a good occasion to clean up your data, deleting parts which will not be used anymore, or to modify others to render them norm compliant.

To modify the bulkmarcimport.pl script needs good knowledge of Perl, and mainly in MARC record manipulation.

The manipulation of MARC records is done by the Perl package MARC::Record.

The description can be found with Perldoc:

perldoc /usr/lib/perl5/site_perl/5.8.3/MARC/Record.pm

(the path is the path of my Mandrake 10.0 machine, it can differ according to your linux/unix distribution)

The same documentation can be found on the net:

http://marcpm.sourceforge.net/MARC/Record.html

and

http://marcpm.sourceforge.net/MARC/Field.html

MARC::Record allows any manipulation directly on the record. It's only flaw is the complex interface which is a direct consequence of the complexity of the iso2709 norm.

The lists of authorized values allow to define a list of possible values for a given subfield. This is especially wanted for subfields where only a well defined and limited set of normalized values are allowed for input.

Examples:

Here follows the example of how to activate the list of language codes.

As soon as this has been done, the subfield isn't a free text zone anymore, but a drop-down list of pre-defined values.

$KOHA/misc/migration_tools/buildLANG.pl

I will not explain to you the nature of authority records according to the UNIMARC norm.

Koha is able to manage authorities in the MARC (UNIMARC or another MARC declination) format. The following section tells how to parametrize the different categories of thesaurus/authorities, and how to migrate a thesaurus.

Koha >> Parameters >> Biblio framework (MARC structure)

Plugins are supplementary modules which can be used to treat data in different ways so as to ease cataloguing of bibliographic (or authority) records.

For example, for UNIMARC, there are plugins facilitating:

[200a ][ / 200b]

200$a => ISBN
200$b => Publisher
200$c (repeatable) => Series

I have some routines to migrate a Texto database. They are not available here as each Texto installation has it's own list of fields.

But the use of a Texto file can be summarized as follows:

$/ = "";
while (<AJOUT_PILOTE>)
{
    my @fichier = split/\n/,$_; 
# print "taille fichier".$#fichier."\n";
    foreach my $ligne (@fichier) {
# on examine le contenu de la fichier
if ($ligne eq "") {# separateur de fichiers

}

my @mots = split(/\./, $ligne);
$mots[0] =~ s/ //g;
$last=$mots[0] if $mots[0];
$resul{$last}.=$mots[1]." ";
    }

# ok, on a l'enregistrement, on construit le MARC::Record.
my $newRecord = MARC::Record->new();

This part of code will build a hash table for each record. From there an iso2709 record can be created, and thus imported normally.

$resul{ISBN} =~ s/-//g;
my $newField = MARC::Field->new(
  '010','','',
  'a' => $resul{ISBN},
);
$newRecord->insert_fields_ordered($newField);
$newField = MARC::Field->new(
  '011','','',
  'a' => $resul{ISSN},
);
newField = MARC::Field->new(
  '200','','',
  'a' => $resul{TIT},
  'b' => $resul{TYP},
  'e' => $resul{STI},
  'f' => $resul{AUT},
  'g' => $resul{NOT},
);
$newRecord->insert_fields_ordered($newField);
my ($bibid,$oldbibnum,$oldbibitemnum) = NEWnewbiblio($dbh,$newRecord,'') unless ($test_parameter);

Please not that the above code doesn't treat item records (this is really much too specific a case).