This document describes the database specific additions to the OSV Schema specific to the Malicious Packages repository.

The OSV Schema is a format specification for reporting vulnerabilities in Open Source projects. This format has been chosen to work with existing OSV tooling.

The Malicious Packages repository uses the OSV format to store reports of malicious packages published on open source repositories.

The additions described in this document have two primary functions:

1. Annotating reports
2. Managing the database

To understand how the standard OSV fields are used, see the Contributing Guide for documentation.

Database Specific Format Overview

The underlying OSV format is a JSON-based encoding format. Additions are expressed in the following informal schema. The exact details of each field are elaborated in the next section. All strings contain UTF-8 text.

{
	/* ... other OSV fields ... */
	"database_specific": {
		"iocs": {
			"domains": [ string ],
			"urls": [ string ],
			"ips": [ string ],
			"files": [ {
				"paths": [ string ],
				"note": string,
				"source": string,
				"digests": {
					"md5": string,
					"sha1": string,
					"sha256": string,
					"tlsh": string,
					"ssdeep": string
				}
			} ],
		},
		"malicious-packages-origins" : [ {
			"source": string,
			"sha256": string,
			"import_time": string,
			"modified_time": string,
			"ranges": [ /* OSV range object */ ],
			"versions": [ string ]
		} ],
		/* ... other database specific entries ... */
	}
}

Field Details

iocs fields (under development)

The iocs field is a JSON object that stores indicators of compromise about a given malicious package.

This area of the database specific schema is under active development and subject to change.

iocs.domains[] field

The domains field is a JSON array of containing domains (e.g. example.com) formatted as ASCII strings. Unicode domains must be converted to Punycode.

iocs.ips[] field

The ips field is a JSON array of strings containing either IPv4 or IPv6 addresses or CIDR blocks. For example 10.1.2.3, 10.1.2.0/24, 2001:db8:a0b:12f0::1, 2001:db8:a0b:12f0::1/32.

iocs.urls[] field

The urls field is a JSON array containing arbitrary URLs strings. For each URL a scheme/authority should be present.

If the URL contains only a host (e.g. example.com), then add the host to either domains or ips field instead.

iocs.files[] field

The files field is a JSON array of objects, each describing a single file associated with the malicious package.

This set records individual files that help identify or attribute the malware, each tagged with a source describing where it came from: a file extracted from the published package artifact, a second stage dropped to disk at run time, or a payload that only ever lived in memory. The published package artifact itself (the tarball/wheel) does not belong here — it is already identified by the record's affected.package; this field is for the individual files of interest, not the archive as a whole.

Each object may contain the following fields. An entry must have at least one paths entry or one digest under digests.

iocs.files[].paths field

The paths field is an optional JSON array of strings giving the file names at which the file was observed, relative or absolute — the same content can appear in several places. Each path is limited to 1024 characters. It may be omitted when the file never has a stable path (e.g. an in-memory payload identified only by its digest).

iocs.files[].note field

The note field is an optional free-text string describing the file. It is limited to 512 characters.

iocs.files[].source field

The source field is an optional string recording where the file came from. It must be one of:

• PACKAGE_ARCHIVE — the file is extracted from the published package artifact.
• DROPPED — the file was written to disk at run time (e.g. a second stage retrieved from a C2, or content decoded from data embedded in the archive).
• IN_MEMORY — the file only ever existed in memory and never hit disk.

iocs.files[].digests field

The digests field is an optional JSON object mapping a hash algorithm name to the file's digest under that algorithm. Recognised algorithms are md5, sha1, sha256, tlsh and ssdeep. sha256 is preferred and recommended at a minimum.

• md5, sha1, sha256 — hex-encoded digests (32, 40 and 64 hex characters respectively).
• tlsh — a hex-encoded TLSH fuzzy hash (70 hex characters, optionally prefixed with the T1 version marker), useful for clustering variants of the same payload.
• ssdeep — an ssdeep context triggered piecewise hash, of the form blocksize:hash1:hash2.

Hex digests (md5, sha1, sha256, tlsh) may be supplied in any case and are normalized to lowercase; ssdeep is case-sensitive and left as-is.

malicious-packages-origins fields

The malicious-packages-origins field is a JSON array containing objects that is used internally to manage the malicious packages database. This data has little value to consumers of OSV malware.

Each entry in the malicious-packages-origins array describes one origin OSV report that was ingested and merged together to produce this individual OSV report.

The field is used to help attribute each source and allow for the automated ingestion command (./cmd/ingest) to detect reports that have already been imported. When an origin OSV is ingested an entry will be added to this array. When OSV for the same package are merged together the malicious-packages-origins in each source OSV will be appended together in the merged OSV.

Within each object in the malicious-packages-origins array, the source field contains a key identifying the specific source where an origin OSV was found. This key should be specified in the ./config/config.yaml file.

The sha256 field contains a hash of the origin OSV file using the SHA256 algorithm. The source and sha256 fields tuple should be a universally unique identifier of an origin OSV file.

For origins that have an OSV id set, the modified_time and id are stored as well to allow for updates to the be detected in the OSV.

The import_time indicates when the original OSV report was imported into the malicious packages repository.

Finally, the ranges and versions fields are copies of the affected[0].ranges and affected[0].versions from the origin OSV.

malicious-packages-origins[].source field

The source field contains a string identifying the specific source that contributed the origin OSV. Can only contain numbers, lowercase characters a-z, and dashes (regexp: ^[a-z0-9-]+$). The source field must be present.

malicious-packages-origins[].sha256 field

The sha256 field is a string containing a SHA256 hash of the origin OSV serialized as a hexadecimal string. Valid characters are 0-9, a-f. The sha256 field must be present.

malicious-packages-origins[].import_time field

The import_time field gives the time the origin OSV should be considered to have been ingested, as an RFC3339-formatted timestamp in UTC (ending in "Z"). The import_time field should be present.

malicious-packages-origins[].id field

The id field is optional. It stores a copy of the ID field in the origin OSV as a string, if it was present. When used in conjunction with modified_time it can be used to identify updates to the origin OSV that need to be consumed.

malicious-packages-origins[].modified_time field

The modified_time field stores a copy of the modified time from the origin OSV, as an RFC3339-formatted timestamp in UTC (ending in "Z"). The modified_time field should be present.

malicious-packages-origins[].ranges field

The ranges field is optional. It is a copy of the ranges field from the affected[].ranges field in the origin OSV.

See https://ossf.github.io/osv-schema/#affectedranges-field.

malicious-packages-origins[].versions field

The versions field is optional. It is a copy of the versions field from the affected[].versions field in the origin OSV.

See https://ossf.github.io/osv-schema/#affectedversions-field.