METHA 1 "JANUAR 2017" "Leipzig University Library" "Manuals"

August 28, 2026 · View on GitHub

NAME

metha - harvest OAI-PMH conform endpoints

SYNOPSIS

metha command [options] [endpoint]

metha [sync|cat|ls|files|id|stat|migrate|pack|fortune|shim]

Since 0.5 metha is a single binary and each of the commands below is a subcommand of it: metha sync rather than metha-sync. The old names remain, as symlinks to the one binary, and take the flags they always did; they print a deprecation notice on a terminal (silence it with METHA_NO_DEPRECATION=1) and go away in metha 2.0. Spelled as subcommands, long options take two dashes: metha sync --format oai_dc. Spelled under an old name, one dash still works.

metha shim install writes the old names next to the binary, which a go install does not do for you. metha help command documents each one, including stat and migrate, which are not described below.

metha-sync [-format FORMAT, -set SET] endpoint

metha-sync [-dir] endpoint

metha-cat [-format FORMAT, -set SET, -from DATE, -until DATE, -root NAME, -j] endpoint

metha-id [-s] endpoint

metha-ls [-a] [-b]

metha-files [-format FORMAT, -set SET] endpoint

metha-fortune [-1, -k N, -t DURATION, -d]

DESCRIPTION

The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a protocol developed for harvesting metadata descriptions of records in an archive. The specification can be found under https://www.openarchives.org/pmh/.

This tool harvests and caches data, so incremental invocations on the same endpoint are fast.

A list of over 80000 (more or less usable) endpoints can be found here: https://is.gd/UrST8m.

OPTIONS

metha-sync

Harvest an OAI-PMH endpoint into the local cache.

-H value extra HTTP header to pass to requests (repeatable); e.g. -H "token: 123"

-T duration http client timeout (default 30s)

-base-dir string base dir for harvested files (default "$HOME/.cache/metha")

-daily use daily intervals for harvesting

-delay duration sleep between each OAI-PMH request

-dir show target directory

-format string metadata format (default "oai_dc")

-from string set the start date, format: 2006-01-02, use only if you do not want the endpoints earliest date

-hourly use hourly intervals for harvesting

-ignore-http-errors do not stop on HTTP errors, just skip to the next interval

-ignore-unexpected-eof ignore unexpected EOF

-k keep temporary files when interrupted

-list list a selection of OAI endpoints (might be outdated)

-log string filename to log to

-log-errors-to-stderr Log errors and warnings to STDERR. If -log or -q are not given, write full log to STDOUT

-max int maximum number of token loops (default 1048576)

-max-empty-responses int allow a number of empty responses before failing (default 10)

-no-compression store harvested files as plain XML instead of .xml.gz or .xml.zst

-no-intervals harvest in one go, for funny endpoints

-q suppress all output

-r int max number of retries (default 10)

-rate-limit string download rate limit (e.g., '1MB', '500KB', '2.5MB/s', '1024'). If no unit specified, bytes/sec assumed. Set to 0 or empty to disable

-rm remove all cached files before starting anew

-set string set name

-suppress-format-parameter do not send format parameter

-u string basic auth, like: user:password

-until string set the end date, format: 2006-01-02, use only if you do not want got records till today

-v show version

metha-cat

Stream harvested records as XML (or JSON) to stdout.

-base-dir string base dir for harvested files (default "$HOME/.cache/metha")

-format string metadata format (default "oai_dc")

-from string ignore records before this date

-j output json, not xml

-root string root element to wrap records into (default "Records")

-set string set name

-until string ignore records after this date

-v show version

metha-id

Print endpoint metadata (identity, formats, sets) as JSON.

-s show size only

-v show version

metha-ls

List cached endpoints.

-a show full path

-b continue in the presence of errors

metha-files

List file paths of a local harvest.

-base-dir string base dir for harvested files (default "$HOME/.cache/metha")

-format string metadata format (default "oai_dc")

-set string set name

-v show version

metha-fortune

Display a random record description from a random endpoint.

-1 one sentence only

-d debug output

-k int number of endpoints to query in parallel (default 16)

-t duration timeout (default 8s)

EXAMPLES

Show metadata about endpoint in JSON format:

metha-id http://export.arxiv.org/oai2

Harvest data in the default oai_dc format:

metha-sync http://export.arxiv.org/oai2

Harvest data in a specific format:

metha-sync -format arXivRaw http://export.arxiv.org/oai2

Harvest a set in a specific format:

metha-sync -set cs -format arXivRaw http://export.arxiv.org/oai2

Show harvested data:

metha-cat http://export.arxiv.org/oai2

Show harvested data for a given set and format:

metha-cat -set cs -format arXivRaw http://export.arxiv.org/oai2

The options -daily, -ignore-http-errors, -suppress-format-parameter, -no-intervals and -max are used to work around non-standard server implementations.

-no-intervals gives up on incremental harvesting: the endpoint cannot answer a date range, so every run fetches the whole repository again. In the v2 layout each run is stored beside the last one, and only the newest is read, so the cache keeps growing while the data stays right. metha-sync says so once the segments pass 10 GB. Reclaim the space with -rm, which starts the harvest from one copy again - metha will not drop the older ones on its own, since an endpoint that has gone away leaves them as the only copy there is.

INTEGRATION

The metha-cat tool emits valid XML to stdout, which can be fed into XML processing tools like xmllint(1).

metha-cat -from 2018-01-01 http://export.arxiv.org/oai2 | xmllint --format -

To get a list of supported formats from an endpoint:

metha-id http://export.arxiv.org/oai2 | jq -r '.formats[].metadataPrefix'

To get a list of available sets from an endpoint:

metha-id http://export.arxiv.org/oai2 | jq -r '.sets[].setSpec'

To remove a harvest completely, remove the harvest directory:

rm -rf $(metha-sync -dir endpoint)

To list cached endpoints you can combine metha-ls and column formatter:

metha-ls -a | column -t

ENVIRONMENT

Deprecated: The harvesting directory can be controlled by the METHA_DIR environment variable. Use the -base-dir flag instead.

Since metha 0.2.0 the XDG Base Directory Specification is followed.

LIMITATIONS

Endpoints URLs longer than about 200 characters are not supported.

Currently the harvest will be up to the last full day, so there will be latency in the data of at most 24 hours.

BUGS

Please report bugs to https://github.com/miku/metha/issues.

ENDPOINTS

A random sample from https://is.gd/UrST8m

https://www.vjol.info.vn/index.php/tcdhHungVuong/oai http://www.iaa.fadu.uba.ar/ojs/index.php/index/oai https://hkjsw.hkswa.org.hk/index.php/index/oai https://eurjchem.com/index.php/eurjchem https://journal.poligran.edu.co/index.php/gsst http://repository.vtc.edu.hk/do/oai/

Curious about the contents of a random endpoint? Run a harvesting roulette with:

URL=$(shuf -n 1 <(curl -Lsf https://git.io/vKXFv)); metha-sync $URL; metha-cat $URL

Select a random record from a random endpoint and display its description:

metha-fortune

UPGRADE TO 0.2.0

To continue using data harvested with previous metha versions, just rename the cache directory. For example, if you used the default, this would be:

mkdir -p $HOME/.cache; mv $HOME/.metha $HOME/.cache/metha

AUTHORS

SEE ALSO

yaz-marcdump(1), xmllint(1), jq(1), fortune(1)