• development version.

bug fixes

• prepInputs() now establishes Google Drive auth with a verified fallback cascade instead of guessing from gargle options. For a configured identity it no longer assumes "an email is set" means it works; it authenticates and then probes the actual file (drive_get()), accepting an identity only when it can read that resource. Order: a token already loaded (manual drive_auth()) wins; otherwise a configured gargle_oauth_email (the user's personal Drive — common case), then a service-account JSON named by GOOGLEDRIVE_AUTH / GARGLE_SERVICE_ACCOUNT, then anonymous/public. A rung whose prerequisite is absent (no email option, no service-account file) is skipped; an identity that authenticates but cannot read the file is deauthorized before the next is tried, so a service-account token never poisons later googledrive calls. Each trial is silent and non-interactive (no OAuth prompt mid-cascade); one Trying … line per rung is emitted at verbose. This fixes a configured service account (often set globally in .Renviron for a bucket or CI) shadowing the user's own files with a 404 "File not found".

• cliCol() (used by messageColoured() / messagePreProcess()) no longer errors with "non-character object(s)" when a reproducible.messageColour* option holds a colour function (e.g. cli::col_red) rather than a colour-name string; such functions now pass through unchanged.

• Cache(useCloud = TRUE) no longer pages the entire cloud-cache Google Drive folder on every call. driveLs() filtered file names locally, so each lookup asked the API for the whole folder (for an accumulated cache that is thousands of files — "Files retrieved so far: 3500 ...") just to keep the handful sharing one cacheId. Because cache files are named <cacheId>..., the per-cacheId lookup now pushes a server-side name contains '<cacheId>' query to googledrive::drive_ls(), so Drive returns only the relevant files. Listings whose filter is not a plain name prefix (e.g. the .dbFile. metadata sweep used by showSimilar across machines) fall back to the previous full listing.

• prepInputsCOG() (the COG /vsicurl/ fast-path) now shows terra's native progress bar during the windowed remote read, which previously ran silently for minutes. The windowed crop is written through terra's block loop so the bar advances as remote tiles are fetched; it is shown only when verbose > 0.

• A public Google Drive file could still launch an interactive OAuth prompt when a service account was configured via the GOOGLEDRIVE_AUTH environment variable (or any case where "auth is configured" did not mean "auth will actually succeed silently"). The previous logic guessed from gargle options/env whether authentication was possible, then let drive_get() attempt it — but plain drive_auth() does not consult GOOGLEDRIVE_AUTH, so the guess was a false positive that fell through to the prompt. assessGoogle() now attempts authentication non-interactively instead of guessing: it tries the GOOGLEDRIVE_AUTH/GARGLE_SERVICE_ACCOUNT service-account JSON, then a cached user token, with rlang_interactive forced FALSE so a missing token errors quietly rather than prompting; only if nothing usable loads does it deauthorize and read the public file anonymously. Net: a loaded or silently-loadable token (incl. a service account) is used; a public file always resolves with no prompt; reproducible.gdriveNoAuth = TRUE still forces the public path.

• A regression in that no-prompt change: a user who had configured gargle auth (gargle_oauth_email set) but had not yet run drive_auth() was silently downgraded to anonymous — the auth attempt forced rlang_interactive = FALSE unconditionally, so with no cached token it failed quietly and assessGoogle() deauthorized googledrive. That 404'd their private Drive files (e.g. a Drive folder, which has no public-mirror remap) and, because drive_deauth() is a global state change, poisoned later direct googledrive calls in the same session. Now the auth attempt only forces non-interactive when no auth is configured (the public-file case, which must never prompt); when a service account or gargle_oauth_email is configured, the session's own interactivity is respected, so a missing token loads from cache or completes OAuth — no manual drive_auth() step needed.

• A further case of that same regression: when a configured user's quiet auth attempt could not silently load a token — e.g. the cached token was minted with an OAuth client googledrive has since changed (tidyverse-erato → tidyverse-clio), or the session is non-interactive — assessGoogle() still called drive_deauth() and read anonymously, 404ing their private file ("File not found") and poisoning later googledrive calls. .gdrivePrepareAuth() now deauthorizes only when auth is not configured (the public-file reader) or reproducible.gdriveNoAuth = TRUE. A configured user whose quiet attempt fails is left untouched, so the real drive_get()/drive_download() runs its normal auth — completing OAuth in an interactive session, or raising gargle's clear "non-interactive auth" error instead of a misleading 404.

new features

• reproducible.urlRemap manifests may now carry an optional id column (the Google Drive file id; also accepted as googledriveId/googledrive_id/ driveId/gid). It is used as a secondary match, by the id parsed from a Drive url, when the resolved filename is unavailable — e.g. an unauthenticated session that cannot read a Drive file's metadata. With it, a Drive URL whose id is in the manifest is redirected to the (public) mirror by prepInputs() before the Drive metadata lookup, so the download needs no Google authentication at all. Manifests without an id column are unchanged.

• reproducible.urlRemap manifests may now carry an optional type column ("file"/"dir") for directory remaps: a "dir" row maps a Google Drive folder id to a bucket prefix-listing URL (e.g. <base>/?prefix=<key>/&delimiter=/). When preProcess() downloads such a folder, it enumerates the folder's files from that public S3 listing (parsed with base R, no xml2 dependency) and downloads each from its mirror URL, instead of googledrive::drive_ls() — so listing a Drive folder needs no authentication. The folder is also recognised as a directory from the manifest alone, avoiding the drive_get() (auth) probe in isGoogleDriveDirectory(). buckethost::makeMirrorManifest(directories = TRUE) emits such a manifest. Manifests without a type column are unchanged.

bug fixes

• The public-Drive no-auth metadata read no longer prevents a configured user from authenticating. The previous change deauthorized googledrive whenever no token was currently loaded, but "no token loaded" is not the same as "cannot authenticate": a user with gargle_oauth_email (+ gargle_oauth_cache) set, or a service-account JSON, can load a cached token silently. assessGoogle() now only falls back to anonymous access when token-less and gargle has no usable non-interactive auth configured (so drive_auth() can silently load the cached token); reproducible.gdriveNoAuth = TRUE still forces anonymous.

• A failed Google Drive access during prepInputs() / preProcess() now reports the full, browser-pasteable URL instead of only the bare fileId. Previously a googledrive::drive_get() failure surfaced e.g. File not found: 13-atqi_7ogRPIFxOoJZoUDYdQCJ5-a_u., which cannot be opened in a browser to check the file exists / is shared. assessGoogle() now wraps the metadata read and re-raises with https://drive.google.com/file/d/<id> (or the original Drive URL), keeping the underlying error detail (e.g. the 404 reason).

• A public Google Drive file no longer triggers an interactive OAuth prompt ("Is it OK to cache OAuth access credentials ...") during prepInputs() / preProcess() when no googledrive token is loaded. The no-auth download path (added previously) was defeated by the metadata read in assessGoogle(): googledrive::drive_get() with no cached token launches interactive OAuth even for an "Anyone with the link" file, and that read happens before the download. assessGoogle() now deauthorizes googledrive for that read when there is no token (the typical "cloud reader" case) or when reproducible.gdriveNoAuth = TRUE, so a public file's metadata resolves anonymously via an API key. A loaded token (a "cloud writer", who needs auth to write a shared cloud cache) is left intact; in the edge case of gdriveNoAuth = TRUE with a token present, the token is restored after the read.

new features

• reproducible.urlRemap now accepts the mirror manifest directly, not only a pre-built function. It may be set to a function(url, filename) (as before, e.g. via makeUrlRemap()), a data.frame manifest with filename/url columns, or a length-one character path/URL to a CSV with those columns. For the data.frame/CSV forms reproducible builds the remap function internally (once, then cached), so a novice can simply write options(reproducible.urlRemap = read.csv("manifest.csv")) without calling makeUrlRemap(). An invalid value is ignored with a warning, so it can never break a download.

bug fixes

• File-backed objects (e.g. a terra SpatRaster) are now restored portably when a cache entry is shared between machines or users (notably a cloud cache). A file-backed raster embeds an absolute path to its backing .tif; previously, when another user retrieved such an entry, Cache() tried to recreate that path on their machine — e.g. cannot create dir '/home/<producer>' ... Operation not supported followed by [rast] file does not exist. Three defects in the existing relative-path machinery caused this: relativeToWhat() used an inverted "is the file under this anchor?" test (so a raster even one directory below the working directory was stored with its absolute path); unwrapSpatRaster() reused the producer's embedded path on the normal load path instead of the stored relative tags; and remapFilenames() had no safe fallback for an unresolved/absolute path. Now the backing file is stored relative to the most specific matching anchor and rebuilt under the receiver's anchor of the same name; when no anchor resolves, the object is made self-contained under the receiver's cachePath rather than the producing machine's absolute path.

• New option reproducible.fileBackedAnchors — a named list of project "anchor" directories (e.g. SpaDES paths(sim)) consulted at both cache save and load so file-backed objects can be stored relative to a semantic, machine-independent anchor (e.g. inputPath) and restored to the equivalent location on another machine. See ?reproducibleOptions.

• getRelative() no longer returns "NA/<basename>" when a path is (or is fully contained in) relativeToPath; it now correctly returns ".". The old result came from (max(id) + 1):length(a) counting backwards when the path matched relativeToPath exactly, producing file.path(NA, last). This corrupted the relativized paths stored by SpaDES.core::saveSimList() and, now that file-backed objects anchor to those paths, would otherwise surface as an NA path segment when restoring a file-backed object.

new features

• Google Drive files that are shared "Anyone with the link" can now be downloaded by prepInputs()/preProcess() without authentication. This happens automatically in three cases: (1) the supplied url is the public web-download form (e.g. https://drive.google.com/uc?export=download&id=<ID>); (2) no googledrive token is loaded — in which case the file's metadata could only have been read anonymously, which is proof it is public, so the public endpoint is used silently instead of failing with a "no token" error; or (3) the new option reproducible.gdriveNoAuth is TRUE. An authenticated download that fails (e.g. an expired token) also silently falls back to the public endpoint, surfacing the original auth error only if the file turns out not to be public. Large files that return Google's "can't scan for viruses" interstitial are handled by parsing and resubmitting the one-time confirm token. Authenticated workflows (token loaded) are unaffected.

• Download progress is now visible in non-interactive / logged sessions. httr2::req_progress() draws a cli progress bar that is silent when !cli::is_dynamic_tty() (logged runs, CI, a SpaDES simInit) and writes straight to the terminal, so a large preProcess()/prepInputs() download there produced no output until it finished. In those sessions the single-stream download now streams the body itself (httr2::req_perform_connection()) and reports progress through messagePreProcess() -- e.g. downloaded 45 Mb / 320 Mb (14%) | 12 Mb/s -- which the calling app (e.g. SpaDES.core's logger) timestamps. The cadence is set by the new option reproducible.downloadProgressInterval (default 2 seconds). In a dynamic terminal the native in-place cli bar is unchanged.

• New diagnostic option reproducible.preDigestDump (and reproducible.preDigestDumpPattern) to dump the full element-by-element preDigest (name = hash) that produces each Cache() cacheId. Unlike showSimilar (closest prior call only), dryRun, or verbose, it covers every Cache() call -- including ones built deep inside other packages (e.g. SpaDES.core events) -- so two machines' dumps can be diffed to find exactly what splits a cacheId across machines/OSs (e.g. a cloud cache that will not share). TRUE messages each call's sorted list; a directory path writes one preDigest_<functionName>[_<n>].txt per call. See ?reproducibleOptions.

• showSimilar (the reproducible.showSimilar option, also used by dev mode and dryRun) is now cloud-aware. When useCloud is active, Cache() previously compared the current call only against the local cache, so similar artifacts cached by other machines sharing the same cloudFolderID were never reported. It now downloads the small per-cacheId metadata files (.dbFile.*) from the cloud folder, folds them into the local cache listing, and follows the normal showSimilar path. Only metadata files are fetched (not the cached objects), cacheIds already present locally are skipped, and each remote metadata file is itself wrapped in Cache() (keyed by its cacheId) so the many Cache() calls in a single run (e.g. a module's .inputObjects) do not re-download the same .dbFile repeatedly across calls or sessions. That memo lives in a dedicated cloudMeta sub-cache, so it does not bloat the main cache's showCache() scans.

• New option reproducible.digestVersion — a single integer that selects the cacheId (digest) algorithm, replacing the per-version booleans reproducible.digestV3/reproducible.digestV4 (which are still honoured when digestVersion is unset). It defaults to 4, a platform-stable digest of sf and SpatVector objects: geometry is the numeric vertex matrix with coordinates rounded to a fixed precision (plus the geometry type), and attributes are kept in feature order with columns sorted locale-independently. The same vector data therefore produces the same cacheId on Windows, macOS and Linux (digest version 3 could differ across operating systems, preventing shared/cloud caching of these objects), and an sf object and its SpatVector equivalent now digest identically. Because version 4 is the new default, the cacheId of every sf/SpatVector object differs from the reproducible package v3.1.1 and earlier, so cached results that involved such objects are recomputed once under the new algorithm. Set options(reproducible.digestVersion = 3) to keep the previous behaviour and avoid that one-time invalidation. Requires the \pkg{terra} package. See the digestVersion entry in ?reproducibleOptions for the full list of versions.

• Digest version 4 also realizes "deferred-string" ALTREP character vectors before hashing, so character content digests identically regardless of its internal representation. A deferred string (e.g. produced by rbind-ing many data.frames, as.character() of a factor, etc.) serializes differently from its realized form — and differently across R versions/platforms — which could give the same character content a different cacheId on, e.g., Linux vs Windows (seen as a module's parameter table splitting the .inputObjects cacheId). The realization is a no-op for already-materialized vectors, so existing cacheIds are unchanged.

• Downloads can now be transparently redirected to faster mirrors and fetched in parallel. Two cooperating, opt-in features:

  • URL remap hook — a new option reproducible.urlRemap accepts a function function(url, filename) that is consulted in the download path once the target filename is resolved (for Google Drive URLs, after the drive_get() lookup). It may return an alternative URL to download from instead — typically a public mirror that supports HTTP Range requests. Returning NULL/the original URL leaves behaviour unchanged, and a remap that errors is ignored (with a warning) so it can never break a download. A new exported helper makeUrlRemap(manifest) builds such a function from a data.frame with filename and url columns (matching on basename). This is the opt-in switch for the faster download path; with the default NULL, nothing changes.
  • Parallel ranged downloads — once opted in via reproducible.urlRemap (with no remap set, downloads are always single-stream), a download that resolves to an HTTPS URL advertising Accept-Ranges: bytes and larger than reproducible.parallel.threshold (default 10 MiB) is fetched as reproducible.parallel.streams (default 48L) concurrent byte-range requests via curl, then reassembled. The result is byte-identical to a single-stream download, so checksums are unaffected. A part that drops mid-transfer is retried individually (not a full re-download); only if a part still cannot complete after a few attempts does it fall back transparently to a single stream (also when ranges are unsupported). Set reproducible.parallel.streams = 1L to force single-stream downloads. On networks that shape bandwidth per-connection this is dramatically faster: in one test a 6.1 GB file dropped from ~75 minutes (single-stream) to ~2 minutes (48 streams from a Range-capable mirror).

• prepInputs() and preProcess() now keep a record of every file and web address (URL) they download. By default, each download is saved as a permanent note on the matching cache entry, which you can look up later with showCache(userTags = "reproducible.url"). Set options(reproducible.urlLog = TRUE) to also keep an in-memory list for the current session, which you can read with prepInputsLog() and empty with clearUrlLog(). Set options(reproducible.urlLog = FALSE) to turn the recording off. See ?prepInputsLog and ?reproducibleOptions.

behaviour changes

• postProcessTo() is now faster and uses less memory on large rasters. A new option reproducible.terraMemmax (default 2, in GB) sets how much memory terra is allowed to use per raster during the call; the previous setting is restored when the call finishes. In one test with a 1.8-billion-cell output, this was about 45% faster and used about one-third of the memory of the previous default. Set the option to NULL to turn the cap off. If you have already set terraOptions(memmax) yourself, that setting is left alone.

• postProcessTo() now keeps categorical (factor) rasters categorical. When the input is a factor raster and you have not supplied method or datatype, the projection step uses nearest-neighbour (so no in-between values are invented) and the output is written with the same data type as the input (for example, INT1U stays INT1U instead of being promoted to FLT4S, which would make the file four times larger and lose the link to the category labels). Anything you pass for method or datatype is respected as before.

• options("reproducible.gdalwarp") is removed. The option was a switch for an experimental alternative dispatch in postProcessTo() that used sf::gdal_utils("gdalwarp") directly. The branch behind the switch had been fully commented out for some time; the live code path was the same whether the option was TRUE or FALSE. Any existing options(reproducible.gdalwarp = ...) calls in user code are now silently ignored and can be deleted. options("reproducible.gdalwarpThreads") (the unrelated thread-count knob for detectThreads()) is unaffected.

• options("reproducible.cachePath") is no longer pre-set to a session-tempdir path when the package is loaded; the default is now NULL. The first call to a user-facing entry point (Cache(), clearCache(), showCache(), keepCache(), ...) resolves it lazily via .checkCacheRepo(). If still unset at that point, the option is set to .reproducibleTempCacheDir() for the rest of the session. This lets project-setup layers (e.g. SpaDES.project::setupProject()) detect "unset" cleanly and stops every R session from silently committing to a non-persistent tempdir cache. Users who set options(reproducible.cachePath = ...) explicitly (in their .Rprofile, in a setup script, or via withr::local_options()) see no change.

• options("reproducible.timeout") default raised from 1200 (20 min) to 12000 (~3.3 h). The previous default caused failures on large (multi-GB) downloads over slow or congested links well before the transfer could complete; the new default keeps the same wall-clock safety net but at a scale appropriate for the file sizes typically handled by prepInputs/preProcess. Users who set options(reproducible.timeout = ...) explicitly see no change.

bug fixes

• A best-effort cloud cache upload no longer aborts the run. When caching an object whose stored form does not include every file CacheStoredFile() predicts from Filenames() (e.g. a cached simList that references its SpatRaster backends rather than copying them under the cacheId), cloudUploadFromCache() previously stop()ed with "File(s) to upload are not available" -- crashing a long, already-locally-saved run during the upload step. It now uploads the files that are present, warns about any it skips, and never errors (the local cache is intact regardless).

• The single-stream preProcess/prepInputs download no longer hangs for hours on a slow or flaky connection. It was setting curl's connecttimeout (the cap on establishing a connection) to reproducible.timeout — the overall download budget, which defaults to 12000 seconds (3.3 h). A stalled TLS handshake (e.g. a transient SSL_connect failure to opendata.nfis.org) therefore froze the session for many minutes before erroring. The connect timeout is now a short, dedicated cap, new option reproducible.connecttimeout (default 30L seconds), mirroring reproducible.parallel.connecttimeout for the parallel ranged path; reproducible.timeout still governs the overall download.

• Parallel ranged downloads are now used only for URLs that the reproducible.urlRemap hook actually redirected to a mirror, matching the documented intent. Previously, once a remap hook was set, the parallel path engaged for any range-capable URL — including direct origin servers that were never redirected. Some such servers (e.g. opendata.nfis.org) cap concurrent connections per IP, so most of the parallel streams stalled with 0 bytes and timed out, making the download far slower than a single stream before eventually falling back. Two changes: (1) a non-redirected URL now downloads single-stream directly; (2) a redirected mirror that nonetheless caps concurrency is detected on the first attempt and falls back to a single stream immediately instead of grinding through every retry. The new threshold for (2) is reproducible.parallel.minConcurrentFrac (default 0.25; 0 disables it).

• Cloud caching no longer silently invents a Google Drive folder when none is specified. Previously, Cache(useCloud = TRUE) with no cloudFolderID (and no options(reproducible.cloudFolderID)) derived a folder name from the local cache path and created/used it. That derived name differs from machine to machine, so two machines computing the identical cacheId each read/write their own cloud folder and never share — the object is recomputed and re-uploaded on every machine, silently. Now:

  • if no cloudFolderID is set (neither the argument nor the option), cloud caching is skipped (local cache only) with a one-time message, since a NULL cloudFolderID means "no cloud target", not "make one up";
  • a cloudFolderID argument of NULL now falls back to options(reproducible.cloudFolderID) (the documented default) before this check, so a globally-set option is honoured;
  • when a cloudFolderID is supplied but cannot be resolved on Drive (not found, or not accessible to the authenticated account), a warning is now emitted (previously silent) explaining that the supplied folder was not used and how to fix it (pass the same accessible Drive folder id on every machine). To share a cloud cache across machines, set the same explicit folder on each, e.g. options(reproducible.cloudFolderID = googledrive::as_id("<id>")).

• A direct preProcess() call no longer prints the target/fun guessing messages ("targetFile was not specified...", "Trying fun on ...", "More than one possible files to load... Picking the last one..."). Because preProcess() never loads the object into R (it only returns file paths), those messages were misleading. They are still shown when preProcess() is called from prepInputs() (where a load follows) and are always suppressed when fun = NA. The returned targetFile/fun are unchanged.

• prepInputs/preProcess no longer error with missing value where TRUE/FALSE needed in the remote hash check when the remote source advertises no file size (e.g. a server with no content-length header). The size comparison now treats a missing remote size as "unknown" and falls through to the normal hash/download path.

• Parallel ranged downloads (the opt-in reproducible.urlRemap path) no longer fail on Windows, where opening all reproducible.parallel.streams (e.g. 48) connections at once was refused at connection time, so almost every part failed and the download fell back to a single stream. The number of simultaneous connections is now capped (the file is still split into many small parts for cheap retries, but only some download at once); the new option reproducible.parallel.maxConnections controls this and defaults to parallelly::availableCores() - 1 (or parallel::detectCores() - 1 when the Suggested parallelly package is absent). The per-part failure reason (from curl) is now reported on each retry and on fallback, instead of being silently discarded. A new option reproducible.parallel.connecttimeout (default 30 seconds) sets the per-connection establishment timeout; this was previously mis-derived from reproducible.timeout and could collapse to ~1 second if that option was lowered.

reproducible 3.1.1

bug fixes

• fixed a test failure (test-destinationPathShared.R) on file systems whose inode numbers exceed .Machine$integer.max: the inode helper now compares inode numbers as strings rather than coercing to integer. No user-facing change; the package code is unaffected.

reproducible 3.1.0

new features

• prepInputsCOG: new fast-path inside prepInputs for remote tiled GeoTiff files (including Cloud Optimized GeoTiffs). When the url is HTTP(S) and at least one of to, cropTo, or maskTo is supplied, only the spatial window of interest is fetched via GDAL's /vsicurl/ — no full-file download. The windowed SpatRaster is returned to the normal postProcess pipeline for crop/mask/write. The fast-path can be disabled with options(reproducible.useCOG = FALSE).

enhancements

• The options reproducible.inputPaths and reproducible.inputPathsRecursive have been renamed to reproducible.destinationPathShared and reproducible.destinationPathSharedRecursive respectively (matching the prepInputs naming family). The old names remain fully functional as backwards-compatible aliases: if reproducible.destinationPathShared is NULL and reproducible.inputPaths is set, the old value is used automatically (with a deprecation message). Update your code by replacing options(reproducible.inputPaths = ...) with options(reproducible.destinationPathShared = ...) at your convenience.

• alsoExtract in prepInputs/preProcess now accepts regex patterns in addition to exact filenames. For example, alsoExtract = "CMD_sm|CMD_sp" will extract all archive members whose name matches that regular expression. The expansion is performed against the archive's file listing using grep(): if an element is a literal match it is kept as-is; otherwise it is treated as a pattern. Special sentinel values ("similar", "none", NA) are not affected. The expansion happens before file extraction whether or not the archive was already present before the call.

• preProcess now skips downloading when a local copy already exists and matches the remote version, even if it was never recorded in CHECKSUMS.txt for the current destinationPath. There is also a new <cacheId>.hash file placed alongside the cached repository files, that is a simpler mechanism than the CHECKSUMS.txt, i.e., one file, one hash. This was implemented because there were too many edge cases that were difficult to handle when there is a single CHECKSUMS.txt file. Nevertheless, the CHECKSUMS.txt file is still used if it is present, and the <>..hash file is absent, so a user can manually place a known CHECKSUMS.txt file into a directory as "the canonical version". The new pp_remote_hash_check stage fetches remote metadata (ETag / content-length for HTTP; md5Checksum / size for Google Drive) and compares against the local file. A stored .hash file is checked first; if absent, file-size equality is used as a fast proxy and the hash is persisted for future runs. This fixes the common cluster pattern where an archive lives in options("reproducible.inputPaths") but has not yet been checksummed for the current run-specific destinationPath.

  Design note — two-layer caching strategy: The remote hash check and the local CHECKSUMS.txt are complementary, not competing. Remote metadata (ETags, content-length, Google Drive md5Checksum) solves the bootstrapping problem — confirming a file is correct before it has ever been checksummed locally. CHECKSUMS.txt then takes over for all subsequent runs: it requires no network round-trip, works on compute nodes without internet access, and is content-addressable (survives URL changes). Note that HTTP ETags are not universally content-hash-based — many servers derive them from inode + mtime, making cross-server or post-migration comparisons unreliable — so the remote check is intentionally a best-effort shortcut rather than a replacement for the local checksum record.

• showCache (when useDBI = FALSE) now has lazy memory caching. This is relevant for very large caches (e.g., >10,000 entries). Cache() now lazy-spawns the showCache async background scan against the cachePath the call actually uses. With the lazy spawn, the first Cache() call in simInit/spades kicks off the fork, which then runs to completion in parallel with the simulation; subsequent showCache() calls return in ~1 second. The spawn helper is idempotent (~10 us per call).

• New exported helper prepopulateCacheAsync(cachePath) lets workflows kick off the async scan explicitly (e.g. early in setupProject()) so the fork has even more wall-clock time to complete.

• Lots of new unit tests for new features, and to cover edge cases that were slipping through.

bug fixes

• Cache(omitArgs = TRUE) now drops every captured argument from the cache digest, so the digest depends only on FUN itself (the function value -- including its body, so source edits still bust the cache) and .cacheExtra. Useful when a developer wants the cache key to be insensitive to the function's inputs and pin freshness via .cacheExtra (e.g. a quoted reference to runtime state) instead of enumerating every input or every argument to omit. Character-vector omitArgs = c("a", "b") still works as before.

• downloadRemote: the "before" snapshot of destinationPath taken before evaluating dlFun now uses recursive = TRUE to match the "after" snapshot. Previously, the non-recursive snapshot omitted files that were already present in subdirectories of destinationPath, so the setdiff() of after vs. before classified those pre-existing subdirectory files as newly created. They then propagated as downloadResults$destFile, triggered the "already exists at . Use overwrite = TRUE?" stop later in the function, and surfaced as a confusing error mentioning unrelated stashed files (e.g. shapefile pieces extracted by an earlier prepInputs call into the same reproducible.inputPaths/reproducible.destinationPathShared).

• Cache(useCloud = ...) now accepts two character values, intended for separating developer and user roles when sharing a cloud-cache folder: "push" is equivalent to TRUE (developer role -- bidirectional; downloads on a cloud hit, uploads on a miss); "pull" is read-only (user role -- downloads on a cloud hit, never uploads). When "pull" is set and the local cache already has the object, the Google Drive listing is not fetched at all (the cloud is consulted only after a local miss). An invalid character value now errors at the front door of Cache() rather than silently being treated as FALSE.

• prepInputs/.guessAtTargetAndFun: no longer auto-selects OS-injected archive metadata when targetFile is unspecified. Previously, a Mac-created zip containing both foo.shp and __MACOSX/._foo.shp could pick the AppleDouble copy and fail to load. Filtering covers macOS (__MACOSX/*, ._* AppleDouble, .DS_Store) and Windows (Thumbs.db, desktop.ini). An explicit targetFile pointing at one of these is still honored.

• pp_remote_hash_check: now skips URLs that are not http:// or https://. Previously, file:// URLs would attempt a HEAD-style metadata fetch and then call makeRemoteHashFile, whose URL-to-filename mapping only strips the https?:// prefix; on Windows the resulting .hash filename retained file: and the drive-letter colon, which is not a legal Windows path character.

• prepInputs/preProcess: when called with dlFun only (no url, targetFile, or archive), the file produced by dlFun is now treated as the source of truth. Previously, runChecksums would still scan getOption("reproducible.inputPaths") and, with no canonical filename to look up, Checksums() listed every file in the stash and matched any of them against the stash's CHECKSUMS.txt. A non-empty match silently redirected destinationPath to the stash and caused prepInputs to load an unrelated file (e.g., a previously stashed shapefile instead of the GADM .rds produced by geodata::gadm()).

• prepInputs/preProcess: dlFun = pkg::fn(args) (a function call passed directly, without quote()-wrapping) is once again kept as a deferred call object instead of being eagerly evaluated. The previous fix for dlFun = if (cond) fn else NULL broke this canonical usage by forcing the lazy promise for all non-quote() expressions. The new logic keeps function-call expressions (fn(args), pkg::fn(args), pkg:::fn(args)) deferred while still evaluating control-flow expressions and bare symbols.

• prepInputsCOG now requires a GeoTiff-style URL extension (.tif, .tiff, .cog, .gtiff) before attempting a /vsicurl/ read. Previously, any HTTP(S) URL combined with a spatial subsetting argument would trigger the fast-path, producing a confusing GDAL Error 4 ... not recognized as being in a supported file format warning when the URL pointed to an archive (e.g. .zip, .tar.gz).

• lockFile now uses checkPath(..., create = TRUE) instead of a silent dir.create(..., showWarnings = FALSE) when creating the lock-file directory, so a missing or unwritable cache directory (e.g., on a network filesystem) produces a clear error rather than a confusing filelock failure.

• Fix .listFilesInArchive incorrectly returning an empty file list for zip archives where archive::archive() reports size = 0 for every entry (a known metadata-reading issue with certain compression variants such as Deflate64). When all reported sizes are zero but the archive file itself is non-empty, all paths are now included rather than being filtered out.

• Fix spurious "More than one possible files to load" message (and "Picking the last one") printed by preProcess even when fun = NA. When the user explicitly passes fun = NA (meaning: do not load the file into R), .guessAtTargetAndFun now returns immediately without inspecting or messaging about the extracted file list.

• Fix pp_remote_hash_check incorrectly treating a direct .tif (or other non-archive) download as an archive when the remote hash matched. The stage was unconditionally setting ctx$archive <- localFile, which caused downstream pp_extract to run 7z/unzip on the plain raster file. Fix: only set ctx$archive when .isArchive(localFile) is non-NULL.

• Fix spurious preProcess could not extract the files from the archive error when files were already present on disk (e.g. extracted earlier in the same call or found via reproducible.inputPaths). In extractFromArchive, result was computed from the checkSums table before .checkSumsUpdate() was called, so freshly-extracted files had no prior entry and NROW(result) == 0 forced the re-extraction branch even though all(isOK) was TRUE. Fix: compute result after .checkSumsUpdate() so it reflects current disk state. The error was non-fatal (caught by the surrounding try()) but printed an alarming message and wasted effort attempting a zero-file extraction.

• Fix Google Drive download failed: HTTP 401 Unauthorized error that occurred mid-session when downloading multiple tiles via prepInputsWithTiles. The raw access_token string extracted from the gargle/googledrive token expired (1-hour TTL) while tiles were being downloaded. Fix: force a gargle token refresh (via Token2.0$refresh()) before each download_resumable_httr2 call, and retry once on 401 with a fresh token for both the httr2 and curl code paths.

• Fix object 'fun' not found error when Cache(prepInputs, ..., fun = fun, ...) is called with fun as a local variable name. substitute(fun) captured the symbol rather than the value; the symbol was then evaluated in the wrong frame (Cache's internal frame, not the user's). Fix: force the R promise directly (funCaptured <- fun) so resolution happens in the frame where the promise was created (the user's frame), regardless of call depth.

• Fix filelock::lock() "Permission denied" error under high parallelism (30+ workers). Three root causes: (1) deleting the .lock file after unlock() broke mutex correctness — workers blocked on fcntl(F_SETLKW) held the old inode's lock while a fresh caller created a new inode and acquired its own "lock" simultaneously; (2) stale .lock files owned by root (from a prior sudo/root run) caused EACCES at open(O_RDWR|O_CREAT); (3) the tryCatch matched on "Permission denied" which is locale-dependent (varies with LC_MESSAGES) — now matches on "Cannot open lock file" (filelock's fixed C-level prefix). Fix: stop deleting lock files after release; wrap filelock::lock() in tryCatch with a 5-attempt retry loop; match on the locale-independent error prefix.

• postProcess: when 2 large polygon datasets were provided (from and to), the pre-cropping step failed as the buffer was not applied. This has been fixed and the buffer now scales with the size of the polygons.

reproducible 3.0.0

• the package qs removed as an option for CacheSaveFormat. The user can stay with declaring options(reproducible.CacheSaveFormat = "qs"), but it will use qs2. qs is being removed from CRAN;
• MacOS: key fixes for paths that have created longstanding failures;
• clearing out of stale code due to Cache rewrite;
• numerous Issues addressed;
• fixes for vignettes on especially MacOS or older R with respect to terra::project issues;
• improved handling of archives when e.g., archive package is not installed or is not able to deal with the compression algorithm (e.g., Deflate64 from Windows);
• showCache now uses a custom memoising internally. Large Cache repositories (>500GB) were slow to showCache. Because it is memoised, this only affects 2nd and subsequent calls in an R session. The first will still be slower.
• CacheGeo handles a few more cases;
• isGoogleDriveDirectory handles more cases correctly (i.e., works now if it is a Google ID);
• minor methods changes e.g., .wrap and .unwrap get defaults for some arguments;
• near complete rewrite of Cache so it is simpler and more robust. The main function is now 200 lines, instead of almost 700; internals all cleaner and maintainable;
• new option reproducible.leaveOnDisk which is only relevant for postProcess with objects that are sometimes disk-backed and sometimes memory-based (like SpatRaster or Raster). The default terra and raster behaviour (which creates unpredictable behaviour, with the transient compute context being the trigger for one behaviour or another) for these was creating unnecessarily slow postProcessing by bringing the objects to memory, sometimes, and this in turn led to unnecessarily slow Cache behaviour because terra::wrap is very slow for large SpatRaster objects. See ?reproducibleOptions;
• showCache (when useDBI() is FALSE) now uses a type of internal memoising, so it is much faster for large cache databases, after a first time called.
• several formerly unexported functions have been converted to dot functions and are now exported e.g., for use in other packages;
• In addition to full rewrites, numerous simplifications throughout code that is still being used;
• There are sufficient changes to the digesting that a user's Cache repository will likely be all or mostly rerun with these package changes. These changes to digesting were required because of incomplete cases that were being missed (i.e., false positive or false negatives). See more details below;
• many internal changes in the postProcess pipeline where terra functions are used. Now all functions are memory-safe, so will not bring the data to memory;
• Cache previously would bring some objects to memory that don't need to, e.g., SpatRaster from terra. These are now left on disk, as part of the Cache pipeline;
• prepInputsWithTiles: new function. prepInputs can now pass through a different sub-function, prepInputsWithTiles, which can deal with (i.e., upload and download) remote files that are tiled. See ?prepInputsWithTiles;
• new experimental feature: cacheChaining; in cases where there are >1 Cache call within a single function, the cacheChaining will digest the containing function (via sys.function(-1)) to determine whether it is stable between calls. If it is unchanged, then a series of Cache calls can be eligible for chaining, meaning where each subsequent Cache call refrains from digesting an object that was the outcome of a prior Cache call, within the same -- and unchanged from the previous time -- function. This can dramatically speed up computations when Cache needs to digest large objects and the digest step takes a long time;
• drop support for R 4.1 and 4.2;
• attribute that was named "call" has been changed to "callInCache" to avoid newly discovered collision with xgboost package that uses that attribute name;
• many minor bugfixes;
• format replaces cacheSaveFormat as an argument so individual Cache calls can switch backend; this can be useful when e.g., qs (which tends to be faster and smaller files) does not work for all types of objects e.g., xgboost.
• CacheGeo added new cases that are able to be used.
• many edge cases were found that were not correctly Cached. This resulted in 2 major changes:
  • rewrite and simplification of Cache;
  • modified digest of the arguments. These changes are not backwards compatible. Details next.
• Fixes to several ongoing "edge cases" that were difficult to address, mostly focused around deeply nested objects that are file-backed with pointers, such terra::SpatRaster class;
• digest changes include the following fixes:
  • CacheDigest which is used within Cache did not digest the names of the list of arguments passed. This did not affect .robustDigest of a normal list, which keeps the names intact.
  • file-backed objects were not correctly unique in the cache as they did not have the cacheId in the filename; now they will have the cacheId prefixed on the file (so they sort alongside the main cache file). This cacheId prefix is removed on recovery from the cache, overwriting any files with the same name.
  • extracting the functionName from a function had several edge cases did not work; these now work
• To maintain as much compatibility with an other Cache database, while losing the more accurate digesting, a user can set options(reproducible.digestV3 = FALSE). This will keep the behaviour where lists are digested without their names for CacheDigest and Cache. This will not affect the file-backed objects changes described above, which will ignore this option.
• useMemoise would work with file-backed objects, but only if the file-backed object did not change after the caching (the pointer to the file was intact, but the file changed). Now, memoising will copy file-backed information from disk each time it "retrieves a file-backed object from memory". This will result in slower memoising than previously. However, it will be robust to downstream changes to the file.
• new function purgeChecksums to allow user to manually purge a file from CHECKSUMS.txt
• new options to help with backwards compatibility:
  • reproducible.useCacheV3: default = TRUE to use the new Cache source code
  • reproducible.digestV3: default = TRUE to use the new Cache digest algorithms
• maskTo can now use a SpatRaster for the mask
• minor bugfixes

reproducible 2.1.2

• remove PackedStatExtent class, releasing it for terra; reproducible now uses a PackedStatExtent2; this will eventually be replaced by the terra PackedStatExtent when this conflict is removed;
• .robustDigest method for "character" no longer will evaluate character strings as files, by default. A user can force the old behaviour with options(reproducible.testCharacterAsFile = TRUE)). This created unwanted, and inexplicable hanging of a computer, e.g., in a data.frame with thousands of rows of a character vector that represent filenames that existed, but their content was not expected to be digested; it would take possibly hours to digest. To digest files, user must explicitly coerce to "Path" with asPath(x), or fs::as_fs_path as the previous hanging behaviour was surprising and could not be easily diagnosed;
• url in prepInputs can now point to a directory; use alsoExtract to pick files by regular expression;
• improved handling of symlinks in remapFileNames();
• pass terra::project() arguments use_gdal and by_util through projectTo() to terra::project();
• in some cases of downloading a file within preProcess, supplying a user_agent (which happens automatically within the function) would cause the download to fail; now there is some redundancy within dlGeneric that will retry without a user_agent if it detects this issue;

Package dependency changes

• begin transition to use cli; instead of custom messaging functions;
• rm crayon dependency;
• begin to replace httr --> convert to use httr2 for some pieces; transition not complete;

Bugfix

• When forcing cacheId, e.g., in Cache(..., cacheId = "myCacheItem"), myCacheItem was not used. Fixed.
• prepInputs(..., fun = sf::st_read) now works as expected ... like prepInputs(..., fun = "sf::st_read")

reproducible 2.1.0

New

• new family of functions that are called inside postProcessTo that use sf::gdal_utils directly. These are still experimental and will only be activated with options("reproducible.gdalwarp" = TRUE)
• default for gdalMask has changed default for "touches". Now has equivalent for terra::mask(..., touches = TRUE), using "-wo CUTLINE_ALL_TOUCHED=TRUE"
• gdalProject now uses 2 threads, setting "-wo NUM_THREADS=2"; can be changed by user with options("reproducible.gdalwarpThreads" = X); see ?reproducibleOptions
• gdal* functions now address datatype issues
• gdal* defaults to FLT8S if datatype not passed
• makeRelative, makeAbsolute and similar have been created to ease many issues encountered in preProcess

Changes

• showSimilar (e.g., options(reproducible.showSimilar = 1)) now preferentially shows the most recent item in cache if there are several with equivalent matching.
• overhaul of messaging in Cache and prepInputs families; functions are highlighted with a different colour; indent level reflects nesting of both Cache and prepInputs, so it is easier to identify which message goes with which function call.
• preProcess is a lot faster now for large numbers of files; uses CHECKSUMS more effectively and fewer times
• retry now captures its expr so it doesn't need a quote; is like try now.
• showSimilar mechanisms now returns the most recent, if there are >1 similar that are equivalently similar
• if a user is having troubles with googledrive for e.g., large files on spotting connections, instructions for using gdown are provided
• showCache, clearCache now have extra arguments fun, cacheId, and ... now can take any arbitrary tag = value pair. The cacheId argument will be very fast if a user is not using useDBI() is FALSE.
• .wrap and .unwrap can now deal with SpatVectorCollection (a terra class that does not have a wrap/unwrap method in terra)
• ALTREP digesting when using spooky or fastdigest were not stable for integers and factors. There is now a work around in .robustDigest that stabilizes these by expanding them from their ALTREP representation first. Since they will be saved and recovered anyway, this will have little effect.
• .wrap and .unwrap are becoming more mature and can handle many more classes effectively. Methods can still be written, if needed.

Testing

• lots of testing with cacheSaveFormat = "qs", which previously was not reliable especially for environments. With all recent changes to .wrap and .unwrap, these appear stable now and should be able to be used for environments.

Bugfixes

• switchDataType can now correctly switch between gdal formats and terra
• many messaging fixes that were imprecise or missing

reproducible 2.0.12

• re-submission after removal from CRAN

reproducible 2.0.11

Remove dependency

• fastdigest was removed from CRAN and so is removed from here.

reproducible 2.0.10

Bug fixes

• critical bugfixes for file-backed SpatRaster objects

reproducible 2.0.9

Enhancements

• new function isUpdated() to determine whether a cached object has been updated;
• makeRelative() is now exported for use downstream (e.g., SpaDES.core);
• new functions getRelative() and normPathRel() for improved symlink handling (#362);
• messaging is improved for Cache with the function named instead of just cacheId
• messaging for prepInputs: minor changes
• more edge cases for Checksums dealt with, so fewer unneeded downloads
• wrapSpatRaster (wrap for file-backed spatRaster objects) fixes for more edge cases
• postProcessTo can now use sf::gdal_utils for the case of from is a gridded object and to is a polygon vector. This appears to be between 2x and 10x faster in tests.
• postProcessTo does a pre-crop (with buffer) to make the projectTo faster. When both from and to are vector objects, this pre-crop appears to create slivers in some cases. This step is now skipped for these cases.
• Cache can now deal with unnamed functions, e.g., Cache((function(x) x)(1)). It will be refered to as "headless".
• terra would fail if internet was unavailable, even when internet is not necessary, due to needing to retrieve projection information. Many cases where this happens will now divert to use sf.
• Cache can now skip calculating objSize, which can take a non-trivial amount of time for large, complicated objects; see reproducibleOptions()

Bug fixes

• Filenames for some classes returned ""; now returns NULL so character vectors are only pointers to files
• Cache on a terra object that writes file to disk, when quick argument is specified was failing, always creating the same object; fixed with #PR368
• useDBI was incorrectly used if a user had set the option prior to package loading. Now works as expected.
• several other minor
• preProcess deals better with more cases of nested paths in archives.
• more edge cases corrected for inputPaths

reproducible 2.0.8

Enhancements

• minor formatting changes
• sometimes a cache entry gets corrupted. Previously, a message was supplied on how to fix; now this is just tried directly instead of just suggesting a user do it.

Bug fixes

• only use character strings when comparing getRVersion() <= "XXX"
• fixes for assessDataType for categorical (factor) Raster and SpatRaster

reproducible 2.0.7

Enhancements

• Address change in round with R > 4.3.1; now a primitive, that does method dispatch. Failure was identified with unit tests, by Luke Tierney who was making the change in base::round.

Bug fixes

• several identified and fixed (PRs by Ceres Barros, notably, PRs #341, #342, #343). These fix missing argument in a .unwrap call, and missing check in preProcess, when targetFilePath was NULL.
• minor documentation updates

reproducible 2.0.5

Enhancements

• Updates of Copy & new .wrap, .unwrap generics and methods to wrap classes that don't save well to disk as is. This uses the name similar to terra::wrap, but with slight differences internally to allow for SpatRaster objects who are file-backed and must have their files moved when they are unwrapped.
• loadFiles updated for more cases
• convert to using withr throughout testing for cleaning up
• more methods for Filename added, including for Path class
• Cache(..., useCloud = TRUE) had many cases that were not working; known cases are now working. Also, now file from file-backed cases are now placed inside the cacheOutputs folder rather than inside a separate folder (used to be "rasters")

Bugfixes

• several small for edge cases

Dependency changes

• none

reproducible 2.0.4

Enhancements

• reproducible.useFuture now defaults to "multisession"
• updated tests to deal with data.table development branch (#314)
• removed all use of data.table::setattr to deal with "modified compiler constants" issue that was detected during CRAN checks
• Improvements with testing using GitHub Actions

Bugfixes

• preProcess failed when googledrive url filename could be found, but destinationPath was not "."
• normPath had different behaviour on *nix-alikes and Windows. Now it is the same.
• SpatRaster objects if saved to a specific, non relative (to getwd()) path would not be recovered correctly (#316)
• Several other Issues that addressed edge cases for prepInputs and family.

reproducible 2.0.2

Enhancements

• new optional backend for Cache via options(reproducible.useDBI = FALSE) is single data files with the same basename as the cached object, i.e., with the same cacheId in the file name. This is a replacement for RSQLite and will likely become the default in the next release. This approach makes cloud caching easier as all metadata are available in small binary files for each cached object. This is simpler, faster and creates far fewer package dependencies (now 11 recursive; before 27 recursive). If a user has DBI and RSQLite installed, then the backend will default to use these currently, i.e., the previous behaviour. The user can change the backend without loss of Cache data.
• moved raster and sp to Suggests; no more internal functions use these. User can still work with Raster and sp class objects as before.
• preProcess can now handle Google docs files, if type = ... is passed.
• postProcess now uses terra and sf internally (with #253) throughout the family of postProcess functions. The previous *Input and *Output functions now redirect to the new *To* functions. These are faster, more stable, and cover vastly more cases than the previous *Inputs family. The old backends no longer work as before.
• minor functions to assist with transition from raster to terra: maxFn, minFn, rasterRead
• .dealWithClass and .dealWithClassOnRecovery are now exported generics, with several methods here, notably, list, environment, default
• other miscellaneous changes to deal with raster to terra transition (e.g. studyAreaName can deal with SpatVector)
• prepInputs now deals with archives that have sub-folder structure are now dealt with correctly in all examples and tests esp. #181.
• prepInputs can now deal with .gdb files. Though, it is limited to sf out of the box, so e.g., Raster layers inside gdb files are not supported (yet?). User can pass fun = NA to not try to load it, but at least have the .gdb file locally on disk.
• hardLinkOrCopy now uses linkOrCopy(symlink = FALSE); more cases dealt with especially nested directory structures that do not exist in the to.
• many GitHub issues closed after transition to using terra and sf.
• preProcess had multiple changes. The following now work: archives with subfolders, archives with subfolders with identical basenames (different dirnames), gdb files, other files where targetFile is a directory.
• ~40 issues were closed with current release.
• code coverage now approaching 85%
• substantial changes to preProcess for minor efficiency gains, edge cases, code cleaning
• new function CacheGeo that weaves together prepInputs and Cache to create a geo-spatial caching. See help and examples.
• maskTo now allows touches arg for terra::mask
• Spatial class is also "fixed" in fixErrorsIn
• prepInputs and preProcess now capture dlFun, so user can pass unquoted dlFun
• Copy method for SpatRaster, with and without file-backing
• Cache(..., useCloud = TRUE) reworked so appears to be more robust than previously.
• maskTo now works even if to is larger than from
• netCDF works with prepInputs; thanks to user nbsmokee with PR #300.

Dependency changes

• no spatial packages are automatically installed any more; to work with prepInputs and family, the user will have to install terra and sf at a minimum.
• terra, sf are in Suggests
• removed entirely: fasterize, fpCompare, magrittr
• moved to Suggests: raster, sp, rlang
• A normal (minimal) install of reproducible no longer installs DBI, nor does it use RSQLite. All cache repositories database files will be in binary individual files in the cacheOutputs file. If a user has DBI and a SQLite engine, then the previous behaviour will be used.

Defunct

• reproducible.useNewDigestAlgorithm is not longer an option as the old algorithms do not work reliably.

Defunct and removed

• removed assessDataTypeGDAL(), clearStubArtifacts(),
• removed non-exported digestRasterLayer2(); evalArgsOnly(); .getSourceURL(); .getTargetCRS(); .checkSums(), .groupedMessage(); .checkForAuxililaryFiles()
• option("reproducible.polygonShortcut") removed

Non exported function changes

• .basename renamed to basename2

Bugfixes

• Cache was incorrectly dealing with environment and environment-like objects. Since some objects, e.g., Spat* objects in terra, must be wrapped prior to saving, environments must be scanned for these classes of objects prior to saving. This previously only occurred for list objects;
• When working with revdep SpaDES.core, there were some cases where the Cache was failing as it could not find the module name;
• during transition from postProcess (using raster and sp) to postProcessTo, some cases are falling through the cracks; these have being addressed.

reproducible 1.2.16

Dependency changes

• none

Enhancements

• Cache now captures the first argument passed to it without evaluating it, so Cache(rnorm(1)) now works as expected.
• As a result of previous, Cache now works with base pipe |> (with R >= 4.1).
• Due to some internal changes in the way arguments are evaluated and digested, there may be some cache entries that will be rerun. However, in simple cases of FUN passed to Cache, there should be no problems with previous cache databases being successfully recovered.
• Added more unit tests
• Reworked Cache internals so that digesting is more accurate, as the correct methods for functions are more accurately found, objects within functions are more precisely evaluated.
• Improved documentation:
  • Examples were reworked, replaced, improved;
  • All user-facing exported functions and methods now have complete documentation;
  • Added () in DESCRIPTION for functions;
  • Added \value in .Rd files for exported methods (structure, the class, the output meaning);
  • Remove commented code in examples.

Bug fixes

• postProcess now also checks resolution when assessing whether to project
• prepInputs has an internal Cache call for loading the object into memory; this was incorrectly evaluating all files if there were more than one file downloaded and extracted. This resulted in cases, e.g. shapefiles, being considered identical if they had the identical geometries, even if their data were different. This is fixed now as it uses the digest of all files extracted.

Deprecated and defunct

• remove defunct argument digestPathContent from Cache
• options("reproducible.useGDAL") is now deprecated; the package is moving towards terra.

reproducible 1.2.11

Dependency changes

• none

Enhancements

• none

Bug fixes

• fix tests for postProcessTo to deal with changes in GDAL/PROJ/GEOS (#253; @rsbivand)
• fixed issue with masking

reproducible 1.2.10

Dependency changes

• Drop support for R 3.6 (#230)
• remove gdalUtilities, gdalUtils, and rgeos from Suggests
• Added minimum versions of raster and terra, because previous versions were causing collisions.

Enhancements

• all direct calls to GDAL are removed: only terra and sf are used throughout
• prepInputs can now take fun as a quoted expression on x, the object loaded by dlFun in preProcess
• preProcess arg dlFun can now be a quoted expression
• changes to the internals and outputs of objSize; now is primarily a wrapper around lobstr::obj_size, but has an option to get more detail for lists and environments.
• .robustDigest now deals explicitly with numerics, which digest differently on different OSs. Namely, they get rounded prior to digesting. Through trial and error, it was found that setting options("reproducible.digestDigits" = 7) was sufficient for all known cases. Rounding to deeper than 7 decimal places was insufficient. There are also new methods for language, integer, data.frame (which does each column one at a time to address the numeric issue)
• New version of postProcess called postProcessTo. This will eventually replace postProcess as it is much faster in all cases and simpler code base thanks to the fantastic work of Robert Hijmans (terra) and all the upstream work that terra relies on
• Minor message updates, especially for "adding to memoised copy...". The three dots made it seem like it was taking a long time. When in reality, it is instantaneous and is the last thing that happens in the Cache call. If there is a delay after this message, then it is the code following the Cache call that is (silently) slow.
• retry can now return a named list for the exprBetween, which allows for more than one object to be modified between retries.

Bug fixes

• .robustDigest was removing Cache attributes from objects under many conditions, when it should have left them there. It is unclear what the issues were, as this would likely not have impacted Cache. Now these attributes are left on.
• data.table objects appear to not be recovered correctly from disk (e.g., from Cache repository. We have added data.table::copy when recovering from Cache repository
• clearCache and cc did not correctly remove file-backed raster files (when not clearing whole CacheRepo); this may have resulted in a proliferation of files, each a filename with an underscore and a new higher number. This fix should eliminate this problem.
• deal with development versions of GDAL in getGDALVersion() (#239)
• fix issue with maskInputs() when not passing rasterToMatch.
• fix issue with isna.SpatialFix when using postProcess.quosure

reproducible 1.2.8

Dependency changes

• lwgeom now a suggested package

Enhancements

• terra class objects can now be correctly saved and recovered by Cache
• fixErrors can now distinguish testValidity = NA meaning don't fix errors and testValidity = FALSE run buffering which fixes many errors, but don't test whether there are any invalid polygons first (maybe slow), or testValidity = TRUE meaning test for validity, then if some are invalid, then run buffer.
• Change default option to reproducible.useNewDigestAlgorithm = 2 which will have user visible changes. To keep old behaviour, set options(reproducible.useNewDigestAlgorithm = 1)
• minor changes to messaging when options(reproducible.showSimilar) is set. It is now more compact e.g., 3 lines instead of 5.
• added sf methods to studyAreaName

Bug fixes

• A small, but very impactful bug that created false positive Cache returns; i.e., a 2nd time through a Cache would return a cached copy, when some of the arguments were different. It occurred for when the differences were in unnamed arguments only.

reproducible 1.2.7

reproducible will be slowly changing the defaults for vector GIS datasets from the sp package to the sf package. There is a large user-visible change that will come (in the next release), which will cause prepInputs to read .shp files with sf::st_read instead of raster::shapefile, as it is much faster. To change now, set options("reproducible.shapefileRead" = "sf::st_read")

Enhancements

• default fun in prepInputs for shapefiles (.shp) is now sf::st_read if the system has sf installed. This can be overridden with options("reproducible.shapefileRead" = "raster::shapefile"), and this is indicated with a message at the moment this is occurring, as it will cause different behaviour.
• quick argument in Cache can now be a character vector, allowing individual character arguments to be digested as character vectors and others to be digested as files located at the specified path as represented by the character vector.
• objSize previously included objects in namespaces, baseenv and emptyenv, so it was generally too large. Now uses the same criteria as pryr::object_size
• improvements with messaging when unzip missing (thanks to @CeresBarros #202)
• while unzipping, will also search for 7z.exe on Windows if the object is larger than 2GB, if can't find unzip.
• fun argument in prepInputs and family can now be a quoted expression.
• archive argument in prepInputs can now be NA which means to treat the file downloaded not as an archive, even if it has a .zip file extension
• many minor improvements to functioning of esp. prepInputs
• speed improvements during postProcess especially for very large objects (>5GB tested). Previously, it was running many fixErrors calls; now only calls fixErrors on fail of the proximate call (e.g., st_crop or whatever)
• retry now has a new argument exprBetween to allow for doing something after the fail (for example, if an operation fails, e.g., st_crop, then run fixErrors, then return back to st_crop for the retry)
• Cache now has MUCH better nested levels detection, with messaging... and control of how deep the Caching goes seems good, via useCache = 2 will only Cache 2 levels in...
• archive argument in prepInputs family can now be NA ... meaning do not try to unzip even if it is a .zip file or other standard archive extension
• gdb.zip files (e.g., a file with a .zip extension, but that should not be opened with an unzip-type program) can now be opened with prepInputs(url = "whateverUrl", archive = NA, fun = "sf::st_read")
• fun argument in prepInputs can now be a quoted function call.
• preProcess now does a better job with large archives that can't be correctly handled with the default zip and unzip with R, by trying system2 calls to possible 7z.exe or other options on Linux-alikes.

Bug fixes

• Copy generic no longer has fileBackedDir argument. It is now passed through with the .... This was creating a bug with some cases where fileBackedDir was not being correctly executed.
• fixErrors() now better handles sf polygons with mixed geometries that include points.
• inadvertent deleting of file-backed rasters in multi-filed stacks during Cache
• writeOutputs.Raster attempted to change datatype of Raster class objects using the setReplacement dataType<-, without subsequently writing to disk via writeRaster. This created bad values in the Raster* object. This now performs a writeRaster if there is a datatype passed to writeOutputs e.g., through prepInputs or postProcess.
• updateSlotFilename has many more tests.
• prepInputs(..., fun = NA) now is the correct specification for "do not load object into R". This essentially replicates preProcess with same arguments.
• several minor bugfixes
• Copy did not correctly copy RasterStacks when some of the RasterLayer objects were in memory, some on disk; raster::fromDisk returned FALSE in those cases, so Copy didn't occur on the file-backed layer files. Using Filenames instead to determine if there are any files that need copying.

reproducible 1.2.6

Enhancements

• Optional (and may be default soon) -- An update to the internal digesting for file-backed Rasters that should be substantially faster, and smaller disk footprint. Set using options("reproducible.useNewDigestAlgorithm" = 2)
• changed default of options("reproducible.polygonShortcut" = FALSE) as there were still too many edge cases that were not covered.

Bug fix

• fixed an error with rcnst on CRAN
• RasterStack objects with a single file (thus acting like a RasterBrick) are now handled correctly by Cache and prepInputs families, especially with new options("reproducible.useNewDigestAlgorithm" = 2), though in tests, it worked with default also
• RSQLite now uses a RNG during dbAppend; this affected 2 tests (#185).

reproducible 1.2.4

Bug fix

• typo in date

reproducible 1.2.3

Bug fix

• minor url fix

reproducible 1.2.2

New features

• removed several uses of rgeos
• moved paddedFloatToChar to reproducible from SpaDES.core.
• increased code coverage
• Pull in legacy %>% code from magrittr to allow the cached alternative, %C%. With new magrittr pipe now in compiled source code, more of the legacy code is required here.

Bug fixes

• several minor

reproducible 1.2.1

New features

• harmonized message colours that are use adjustable via options: reproducible.messageColourPrepInputs for all prepInputs functions; reproducible.messageColourCache for all Cache functions; and reproducible.messageColourQuestion for questions that require user input. Defaults are cyan, blue and green respectively. These are user-visible colour changes.
• improved messaging for Cache cases where a file.link is used instead of saving.
• with improved messaging, now options(reproducible.verbose = 0) will turn off almost all messaging.
• postProcess and family now have filename2 = NULL as the default, so not saved to disk. This is a change.
• verbose is now an argument throughout, whose default is getOption(reproducible.verbose), which is set by default to 1. Thus, individual function calls can be more or less verbose, or the whole session via option.

Bug fixes

• RasterStack objects were not correctly saved to disk under some conditions in postProcess - fixed
• several minor

reproducible 1.2.0

New features

• postProcess now uses a simpler single call to gdalwarp, if available, for RasterLayer class to accomplish cropInputs, projectInputs, maskInputs, and writeOutputs all at once. This should be faster, simpler and, perhaps, more stable. It will only be invoked if the RasterLayer is too large to fit into RAM. To force it to be used the user must set useGDAL = "force" in prepInputs or postProcess or globally with options("reproducible.useGDAL" = "force")
• postProcess when using the new gdalwarp, has better persistence of colour table, and NA values as these are kept with better reliability
• concurrent Cache now works as expected (e.g., with parallel processing, it will avoid collisions) with SQLite thanks to suggestion here: https://stackoverflow.com/a/44445010
• updated digesting of Raster class objects to account for more of the metadata (including the colortable). This will change the digest value of all Raster layers, causing re-run of Cache
• removed Require, pkgDep, trimVersionNumber, normPath, checkPath that were moved to Require package. For backwards compatibility, these are imported and reexported
• address permanently or temporarily new changes in GDAL>3 and PROJ>6 in the spatial packages.
• new function file.move used to rename/copy files across disks (a situation where file.rename would fail)
• all DBI type functions now have default cachePath of getOption("reproducible.cachePath")
• Cache(prepInputs, ... on a file-backed Raster* class object now gives the non-Cache repository folder as the filename(returnRaster). Previously, the return object would contain the cache repository as the folder for the file-backed Raster*

Dependency changes

• net reduction in number of packages that are imported from by 14. Removed completely: backports, memoise, quickPlot, R.utils, remotes, tools, and versions; moved to Suggests: fastdigest, gdalUtils, googledrive, httr, qs, rgdal, sf, testthat; added: Require. Now there are 12 non-base packages listed in Imports. This is down from 31 prior to Ver 1.0.0.

bug fixes

• fix over-wide tables in PDF manual (#144)
• use file.link not file.symlink for saveToCache. This would have resulted in C Stack overflow errors due to missing original file in the file.symlink
• use system call to unzip when extracting large (>= 4GB) files (#145, @tati-micheletti)
• several minor including projectInputs when converting to longlat projections, setMinMax for gdalwarp results
• Filenames now consistently returns a character vector (#149)
• improvements to file-backed Raster caching to accommodate a few more edge cases

reproducible 1.1.1

New features

• none

Dependency changes

• none

bug fixes

• fix CRAN test failure when file.link does not succeed.

reproducible 1.1.0

New features

• begin to accommodate changes in GDAL/PROJ and associated updates to other spatial packages. More updates are expected as other spatial packages (namely raster) are updated.
• can now change options('reproducible.cacheSaveFormat') on the fly; cache will look for the file by cacheId and write it using options('reproducible.cacheSaveFormat'). If it is in another format, Cache will load it and resave it with the new format. Experimental still.
• new Copy methods for refClass objects, SQLite and moved environment method into ANY as it would be dispatched for unknown classes that inherit from environment, of which there are many and this should be intercepted
• Require can now handle minimum version numbers, e.g., Require("bit (>=1.1-15.2)"); this can be worked into downstream tools. Still experimental.
• Cache will do file.link or file.symlink if an existing Cache entry with identical output exists and it is large (currently 1e6 bytes); this will save disk space.
• Cache database now has tags for elapsed time of "digest", "original call", and "subsequent recovery from file", elapsedTimeDigest, elapsedTimeFirstRun, and elapsedTimeLoad, respectively.
• Better management of temporary files in package and tests, e.g., during downloading (preProcess). Includes 2 new functions, tempdir2 and tempfile2 for use with reproducible package
• New option: reproducible.tempPath, which is used for the new control of temporary files. Defaults to file.path(tempdir(), "reproducible"). This feature was requested to help manage large amounts of temporary objects that were not being easily and automatically cleaned
• Copying or moving of Cache directories now works automatically if using default drv and conn; user may need to manually call movedCache if cache is not responding correctly. File-backed Rasters are automatically updated with new paths.
• Cache now treats file-backed Rasters as though they had a relative path instead of their absolute path. This means that Cache directories can be copied from one location to another and the file-backed Raster* will have their filenames updated on the fly during a Cache recovery. User doesn't need to do anything.
• postProcess now will perform simple tests and skip cropInputs and projectInputs with a message if it can, rather than using Cache to "skip". This should speed up postProcess in many cases.
• messaging with Cache has change. Now, cacheId is shown in all cases, making it easier to identify specific items in the cache.
• Automatically cleanup temporary (intermediate) raster files (with #110).

Dependency changes

• none

bug fixes

• Copy only creates a temporary directory for filebacked rasters; previously any Copy command was creating a temporary directory, regardless of whether it was needed
• cropInputs.spatialObjects had a bug when object was a large non-Raster class.
• cropInputs may have failed due to "self intersection" error when x was a SpatialPolygons* object; now catches error, runs fixErrors and retries crop. Great reprex by @tati-micheletti. Fixed in commit 89e652ef111af7de91a17a613c66312c1b848847 .
• Filenames bugfix related to RasterBrick
• prepInputs does a better job of keeping all temporary files in a temporary folder; and cleans up after itself better.
• prepInputs now will not show message that it is loading object into R if fun = NULL (#135).

reproducible 1.0.0

New features

• This version is not backwards-compatible out of the box. To maintain backwards compatibility, set: options("reproducible.useDBI" = FALSE)
• A new backend was introduced that uses DBI package directly, without archivist. This has much improved speed.
• New option: options("reproducible.cacheSaveFormat"). This can be either rds (default) or qs. All cached objects will be saved with this format. Previously it was rda.
• Cache objects can now be saved with with qs::qsave. In many cases, this has much improved speed and file sizes compared to rds; however, testing across a wide range of conditions will occur before it becomes the default.
• Changed default behaviour for memoising ... because Cache is now much faster, the default is to turn memoising off, via options("reproducible.useMemoise" = FALSE). In cases of large objects, memoising should still be faster, so user can still activate it, setting the option to TRUE.
• Much better SQLite database handling for concurrent write attempts. Tested with dozens of write attempts per second by 3 cores with abundant locked database occurrences.
• postProcess arg useGDAL can now take "force" as the default behaviour is to not use GDAL if the problem can fit into RAM and sf or raster tools will be faster than GDAL tools
• useCloud argument in Cache and family has slightly modified functionality (see ?Cache new section useCloud) and now has more tests including edge cases, such as useCloud = TRUE, useCache = 'overwrite'. The cloud version now will also follow the "overwrite" command.

Dependency changes

• deprecating archivist; moved to Suggests.
• removed imports for bitops, dplyr, fasterize, flock, git2r, lubridate, RcppArmadillo, RCurl and tidyselect. Some of these went to Suggests.

bug fixes

• postProcess calls that use GDAL made more robust (including #93).
• Several minor, edge cases were detected and fixed.

reproducible 0.2.11

Dependency changes

• remove dplyr as a direct dependency. It is still an indirect dependency through DiagrammeR

New features

• new option: reproducible.showSimilarDepth allows for a deeper assessment of nested lists for differences between the nearest cached object and the present object. This greater depth may allow more fine tuned understanding of why an object is not correctly caching
• for downloading large files from GoogleDrive (currently only implemented), if user has set options("reproducible.futurePlan") to something other than FALSE, then it will show download progress if the file is "large".

bug fixes

• Several minor, edge cases were detected and fixed.

reproducible 0.2.10

Dependency changes

• made compatible with googledrive v 1.0.0 (#119)

New features

• pkgDep2, a new convenience function to get the dependencies of the "first order" dependencies.
• useCache, used in many functions (incl Cache, postProcess) can now be numeric, a qualitative indicator of "how deep" nested Cache calls should set useCache = TRUE -- implemented as 1 or 2 in postProcess currently. See ?Cache

bug fixes

• pkgDep was becoming unreliable for unknown reasons. It has been reimplemented, much faster, without memoising. The speed gains should be immediately noticeable (6 second to 0.1 second for pkgDep("reproducible"))
• improved retry to use exponential backoff when attempting to access online resources (#121)

reproducible 0.2.9

New features

• Cache has 2 new arguments, useCloud and cloudFolderID. This is a new approach to cloud caching. It has been tested with file backed RasterLayer, RasterStack and RasterBrick and all normal R objects. It will not work for any other class of disk-backed files, e.g., ff or bigmatrix, nor is it likely to work for R6 class objects.
• Slowly deprecating cloudCache and family of functions in favour of a new approach using arguments to Cache, i.e., useCache and cloudFolderID
• downloadData from Google Drive now protects against HTTP2 error by capturing error and retrying. This is a curl issue for interrupted connections.

Bug fixes

• fixes for rcnst errors on R-devel, tested using devtools::check(env_vars = list("R_COMPILE_PKGS"=1, "R_JIT_STRATEGY"=4, "R_CHECK_CONSTANTS"=5))
• other minor improvements, including fixes for #115

reproducible 0.2.8

New features

• new functions for accessing specific items from the cacheRepo: getArtifact, getCacheId, getUserTags
• retry, a new function, wraps try with an explicit attempt to retry the same code upon error. Useful for flaky functions, such as googldrive::drive_download which sometimes fails due to curl HTTP2 error.
• removed all Rcpp functionality as the functions were no longer faster than their R base alternatives.

Bug fixes

• prepInputs was not correctly passing useCache
• cropInputs was reprojecting extent of y as a time saving approach, but this was incorrect if studyArea is a SpatialPolygon that is not close to filling the extent. It now reprojects studyArea directly which will be slower, but correct. (#93)
• other minor improvements

reproducible 0.2.7

New features

• CHECKSUMS.txt should now be ordered consistently across operating systems (note: base::order will not succeed in doing this --> now using .orderDotsUnderscoreFirst)
• cloudSyncCache has a new argument: cacheIds. Now user can control entries by cacheId, so can delete/upload individual objects by cacheId
• Experimental support within the postProcess family for sf class objects

bug fixes

• mostly minor
• cloudCache bugfixes for more cases

reproducible 0.2.6

Dependency changes

• remove tibble from Imports as it's no longer being used

New features

• remove %>% pipe that was long ago deprecated. User should use %C% if they want a pipe that is Cache-aware. See examples.
• Full rewrite of all options descriptions now in reproducible, see ?reproducibleOptions
• now cacheRepo and options("reproducible.cachePath") can take a vector of paths. Similar to how .libPaths() works for libraries, Cache will search first in the first entry in the cacheRepo, then the second etc. until it finds an entry. It will only write to the first entry.
• new value for the option: options("reproducible.useCache" = "devMode"). The point of this mode is to facilitate using the Cache when functions and datasets are continually in flux, and old Cache entries are likely stale very often. In devMode, the cache mechanism will work as normal if the Cache call is the first time for a function OR if it successfully finds a copy in the cache based on the normal Cache mechanism. It differs from the normal Cache if the Cache call does not find a copy in the cacheRepo, but it does find an entry that matches based on userTags. In this case, it will delete the old entry in the cacheRepo (identified based on matching userTags), then continue with normal Cache. For this to work correctly, userTags must be unique for each function call. This should be used with caution as it is still experimental.
• change to how hashes are calculated. This will cause existing caches to not work correctly. To allow a user to keep old behaviour (during a transition period), the "old" algorithm can be used, with options("reproducible.useNewDigestAlgorithm" = FALSE). There is a message of this change on package load.
• add experimental cloud* functions, especially cloudCache which allows sharing of Cache among collaborators. Currently only works with googledrive
• updated assessDataType to consolidate assessDataTypeGDAL and assessDataType into single function (#71, @ianmseddy)
• cc: new function -- a shortcut for some commonly used options for clearCache()
• added experimental capacity for prepInputs to handle .rar archives, on systems with correct binaries to deal with them (#86, @tati-micheletti)
• remove fastdigest::fastdigest as it is not return the identical hash across operating systems

Bug fixes

• prepInputs on GIS objects that don't use raster::raster to load object were skipping postProcess. Fixed.
• under some circumstances, the prepInputs would cause virtually all entries in CHECKSUMS.txt to be deleted. 2 cases where this happened were identified and corrected.
• data.table class objects would give an error sometimes due to use of attr(DT). Internally, attributes are now added with data.table::setattr to deal with this.
• calling gdalwarp from prostProcess now correctly matches extent (#73, @tati-micheletti)
• files from url that have unknown extension are now guessed with by preProcess (#92, @tati-micheletti)

reproducible 0.2.5

Dependency changes

• Added remotes to Imports and removed devtools

New features

• New value possible for options(reproducible.useCache = 'overwrite'), which allows use of Cache in cases where the function call has an entry in the cacheRepo, will purge it and add the output of the current call instead.

• New option reproducible.inputPaths (default NULL) and reproducible.inputPathsRecursive (default FALSE), which will be used in prepInputs as possible directory sources (searched recursively or not) for files being downloaded/extracted/prepared. This allows the using of local copies of files in (an)other location(s) instead of downloading them. If local location does not have the required files, it will proceed to download so there is little cost in setting this option. If files do exist on local system, the function will attempt to use a hardlink before making a copy.

• dlGoogle() now sets options(httr_oob_default = TRUE) if using Rstudio Server.

• Files in CHECKSUMS now sorted alphabetically.

• Checksums can now have a CHECKSUMS.txt file located in a different place than the destinationPath

• Attempt to select raster resampling method based on raster type if no method supplied (#63, @ianmseddy)

• projectInputs

• new function assessDataTypeGDAL, used in postProcess, to identify smallest datatype for large Raster* objects passed to GDAL system call

  • when masking and reprojecting large Raster objects, enact gdalwarp system call if raster::canProcessInMemory(x,4) = FALSE for faster and memory-safe processing
  • better handling of various data types in Raster objects, including factor rasters

Bug fixes

• Work around internally inside extractFromArchive for large (>2GB) zip files. In the R help manual, unzip fails for zip files >2GB. This uses a system call if the zip file is too large and fails using base::unzip.
• Work around for raster::getData issues.
• Speed up of Cache() when deeply nested, due to grep(sys.calls(), ...) that would take long and hang.
• Bugfix for preProcess(url = NULL) (#65, @tati-micheletti)
• Improved memory performance of clearCache (#67), especially for large Raster objects that are stored as binary R files (i.e., .rda)
• Other minor bugfixes

Other changes

• Deal with new raster package changes in development version of raster package
• Added checks for float point number issues in raster resolutions produced by raster::projectRaster
• .robustDigest now does not include Cache-added attributes
• Additional tests for preProcess() (#68, @tati-micheletti)
• Many new unit tests written, which caught several minor bugs

reproducible 0.2.3

• fix and skip downloading test on CRAN

reproducible 0.2.2

Dependency changes

• Add future to Suggests.

New features

• new option on non-Windows OSs to use future for Cache saving to SQLite database, via options("reproducible.futurePlan"), if the future package is installed. This is FALSE by default.
• If a do.call function is Cached, previously, it would be labelled in the database as do.call. Now it attempts to extract the actual function being called by the do.call. Messaging is similarly changed.
• new option reproducible.ask, logical, indicating whether clearCache should ask for deletions when in an interactive session
• prepInputs, preProcess and downloadFile now have dlFun, to pass a custom function for downloading (e.g., "raster::getData")
• prepInputs will automatically use readRDS if the file is a .rds.
• prepInputs will return a list if fun = "base::load", with a message; can still pass an envir to obtain standard behaviour of base::load.
• clearCache - new argument ask.
• new function assessDataType, used in postProcess, to identify smallest datatype for Raster* objects, if user does not pass an explicit datatype in prepInputs or postProcess (#39, @CeresBarros).

Bug fixes

• fix problems with tests introduced by recent git2r update (@stewid, #36).
• .prepareRasterBackedFile -- now will postpend an incremented numeric to a cached copy of a file-backed Raster object, if it already exists. This mirrors the behaviour of the .rda file. Previously, if two Cache events returned the same file name backing a Raster object, even if the content was different, it would allow the same file name. If either cached object was deleted, therefore, it would cause the other one to break as its file-backing would be missing.
• options were wrongly pointing to spades.XXX and should have been reproducible.XXX.
• copyFile did not perform correctly under all cases; now better handling of these cases, often sending to file.copy (slower, but more reliable)
• extractFromArchive needed a new Checksum function call under some circumstances
• several other minor bug fixes.
• extractFromArchive -- when dealing with nested zips, not all args were passed in recursively (#37, @CeresBarros)
• prepInputs -- arguments that were same as Cache were not being correctly passed internally to Cache, and if wrapped in Cache, it was not passed into prepInputs. Fixed.
• .prepareFileBackedRaster was failing in some cases (specifically if it was inside a do.call) (#40, @CeresBarros).
• Cache was failing under some cases of Cache(do.call, ...). Fixed.
• Cache -- when arguments to Cache were the same as the arguments in FUN, Cache would "take" them. Now, they are correctly passed to the FUN.
• preProcess -- writing to checksums may have produced a warning if CHECKSUMS.txt was not present. Now it does not.
• numerous other minor bugfixes

Other changes

• most tests now use a standardized approach to attaching libraries, creating objects, paths, enabling easier, error resistant test building

reproducible 0.2.1

New features

• new functions:

  • convertPaths and convertRasterPaths to assist with renaming moved files.

• prepInputs -- new features

  • alsoExtract now has more options (NULL, NA, "similar") and defaults to extracting all files in an archive (NULL).
  • skips postProcess altogether if no studyArea or rasterToMatch. Previously, this would invoke Cache even if there was nothing to postProcess.

Bug fixes

• copyFile correctly handles directory names containing spaces.
• makeMemoisable fixed to handle additional edge cases.
• other minor bug fixes.

reproducible 0.2.0

New features

• new functions:

  • prepInputs to aid in data downloading and preparation problems, solved in a reproducible, Cache-aware way.
  • postProcess which is a wrapper for sequences of several other new functions (cropInputs, fixErrors, projectInputs, maskInputs, writeOutputs, and determineFilename)
  • downloadFile can handle Google Drive and ftp/http(s) files
  • zipCache and mergeCache
  • compareNA does comparisons with NA as a possible value e.g., compareNA(c(1,NA), c(2, NA)) returns FALSE, TRUE

• Cache -- new features:

  • new arguments showSimilar, verbose which can help with debugging
  • new argument useCache which allows turning caching on and off at a high level (e.g., options("useCache"))
  • new argument cacheId which allows user to hard code a result from a Cache
  • deprecated arguments: digestPathContent --> quick, compareRasterFileLength --> length
  • Cache arguments now propagate inward to nested Cache function calls, unless explicitly set on the inner functions
  • more precise messages provided upon each use
  • many more userTags added automatically to cache entries so much more powerful searching via showCache(userTags="something")

• checksums now returns a data.table with the same columns whether write = TRUE or write = FALSE.

• clearCache and showCache now give messages and require user intervention if request to clearCache would be large quantities of data deleted

• memoise::memoise now used on 3rd run through an identical Cache call, dramatically speeding up in most cases

• new options: reproducible.cachePath, reproducible.quick, reproducible.useMemoise, reproducible.useCache, reproducible.useragent, reproducible.verbose

• asPath has a new argument indicating how deep should the path be considered when included in caching (only relevant when quick = TRUE)

• New vignette on using Cache

• Cache is parallel-safe, meaning there are tryCatch around every attempt at writing to SQLite database so it can be used safely on multi-threaded machines

• bug fixes, unit tests, more imports for packages e.g., stats

• updates for R 3.6.0 compact storage of sequence vectors

• experimental pipes (%>%, %C%) and assign %<%

• several performance enhancements

reproducible 0.1.4

• mergeCache: a new function to merge two different Cache repositories

• memoise::memoise is now used on loadFromLocalRepo, meaning that the 3rd time Cache() is run on the same arguments (and the 2nd time in a session), the returned Cache will be from a RAM object via memoise. To stop this behaviour and use only disk-based Caching, set options(reproducible.useMemoise = FALSE) .

• Cache assign -- %<% can be used instead of normal assign, equivalent to lhs <- Cache(rhs).

• new option: reproducible.verbose, set to FALSE by default, but if set to true may help understand caching behaviour, especially for complex highly nested code.

• all options now described in ?reproducible.

• All Cache arguments other than FUN and ... will now propagate to internal, nested Cache calls, if they are not specified explicitly in each of the inner Cache calls.

• Cached pipe operator %C% -- use to begin a pipe sequence, e.g., Cache() %C% ...

• Cache arg sideEffect can now be a path

• Cache arg digestPathContent default changed from FALSE (was for speed) to TRUE (for content accuracy)

• New function, searchFull, which shows the full search path, known alternatively as "scope", or "binding environments". It is where R will search for a function when requested by a user.

• Uses memoise::memoise for several functions (loadFromLocalRepo, pkgDep, package_dependencies, available.packages) for speed -- will impact memory at the expense of speed.

• New Require function

  • attempts to create a lighter weight package reproducibility chain. This function is usable in a reproducible workflow: it includes both installing and loading of packages, it can maintain version numbers, and uses smart caching for speed. In tests, it can evaluate whether 20 packages and their dependencies (~130 packages) are installed and loaded quickly (i.e., if all TRUE, ~0.1 seconds). This is much slower than running require on those 20 packages, but require does not check for dependencies and deal with them if missing: it just errors. This speed should be fast enough for many purposes.
  • can accept uncommented name, if length 1.

• remove dplyr from Imports

• Add RCurl to Imports

• change name of digestRaster to .digestRaster

reproducible 0.1.3

• fix R CMD check errors on Solaris that were not previously resolved

reproducible 0.1.2

• fix R CMD check errors on Solaris
• fix bug in digestRaster affecting in-memory rasters
• move rgdal to Suggests

reproducible 0.1.1

• cleanup examples and do run them (per CRAN)
• add tests to ensure all exported (non-dot) functions have examples

reproducible 0.1.0

• A new package, which takes all caching utilities out of the SpaDES package.