@qiwi/masker

Composite data masking utility

• Digest
  • Purpose
  • Status
  • Roadmap
  • Key features
• Getting started
  • Install
  • Default preset
  • Custom pipeline
  • Masking schema
  • CLI
  • Playground
• Integration
  • Console
  • Winston
• Design
• Documentation
• Contributing
• Packages
• License

Digest

Purpose

Implement instruments, describe practices, contracts to solve sensitive data masking problem in JS/TS. For secure logging, for public data output, for internal mimt-proxies (kuber sensitive-data-policy) and so on.

Status

🚧 Work in progress / MVP#0 is available for testing
⚠️ Not ready for production yet

Roadmap

• Implement masking composer/processor
• Introduce (declarative?) masking directives: schema
• Describe masking strategies and add masking utils
• Support logging tools integration

Key features

• Both sync and async API
• Declarative configuration
• Deep customization
• TS and Flow typings

Getting started

Install

With npm:

npm install --save @qiwi/masker

or yarn:

yarn add @qiwi/masker

Default preset

import {masker} from '@qiwi/masker'

// Suitable for most std cases: strings, objects, json strings, which may contain any standard secret keys/values or card PANs.
masker('411111111111111')       // Promise<4111 **** **** 1111>
masker.sync('4111111111111111') // 4111 **** **** 1111

Custom pipeline

import {masker, registry} from '@qiwi/masker'

masker.sync({
  secret: 'foo',
  nested: {
    pans: [4111111111111111]
  },
  foo: 'str with printed password=foo and smth else',
  json: 'str with json inside {"secret":"bar"} {"4111111111111111":"bar"}',
}, {
  registry,           // plugin storage
  pipeline: [
    'split',          // to recursively process object's children. The origin `pipeline` will be applied to internal keys and values
    'pan',            // to mask card PANs
    'secret-key',     // to conceal sensitive fields like `secret` or `token` (pattern is configurable)
    'secret-value',   // to replace sensitive parts of strings like `token=foobar` (pattern is configurable)
    'json',           // to find jsons in strings
  ]
})

// result:
{
  secret: '***',      // secret-key
  nested: { // split
    pans: [ // split
      '4111 **** **** 1111' // pan
    ],  
  },
  foo: 'str with printed *** and smth else',  // secret-value
  // json
  // chunk#1: split, secret-key
  // chunk#2: split, pan (applied to key!)
  json: 'str with json inside {"secret":"***"} {"4111 **** **** 1111":"bar"}'
}

Masking schema

Declare masker directives over json-schema. See @qiwi/masker-schema for details.

import {masker} from '@qiwi/masker';

masker.sync({
  fo: 'fo',
  foo: 'bar',
  foofoo: 'barbar',
  baz: 'qux',
  arr:  [4111111111111111, 1234123412341234]
}, {
  pipeline: ['schema'],
  schema: {
    type: 'object',
    properties: {
      fo: {
        type: 'string',
        maskKey: ['plain']
      },
      foo: {
        type: 'string',
        maskKey: ['plain']
      },
      foofoo: {
        type: 'string',
        maskKey: ['strike'],
        maskValue: ['plain']
      },
      arr: {
        type: 'array',
        items: {
          type: 'number',
          maskValue: ['pan']
        }
      }
    }
  }
})

// result:
{
  baz: 'qux',
  arr: [ '4111 **** **** 1111', '1234123412341234' ],
  '***': 'fo',
  '***(2)': 'bar',
  '******': '***',
}

CLI

npx masquer "4111 1111 1111 1111"
# returns 4111 **** **** 1111

Playground

codesandbox.io/s/qiwi-masker-sandbox-ngrnu

Integration

Console

Override global console methods to print sensitive data free output to stderr/stdout:

import {masker} from '@qiwi/masker'

['log', 'info', 'error'].forEach(method => {
  const _method = console[method]
  console[method] = (...args: any[]) => _method(...args.map(masker))
})

Winston

Create a custom masker formatter, then attach it to your reporter / transport:

const winston = require('winston')
const {masker} = require('@qiwi/masker')

const logger = winston.createLogger({
  levels: winston.config.syslog.levels,
  transports: [
    new winston.transports.Console({
      format: winston.format.combine(
        winston.format((info) => Object.assign(info, masker.sync(info)))(),
        winston.format.json(),
      ),
    })
  ]
})

logger.log({
  level: 'info',
  message: {foo: 'bar', secret: 'foobar', pan: [4111111111111111, 1234123412341234]},
})

// stdout
{"level":"info","message":{"foo":"bar","secret":"***","pan":["4111 **** **** 1111","1234123412341234"]}}

stackoverflow.com/how-to-make-a-custom-json-formatter-for-winston3-logger

Design

Middleware

The masker bases on the middleware pattern: it takes a piece of data and pushes it forward the pipeline. The output of each pipe is the input for the next one. Each pipe is a dual interface data processor:

export interface IMaskerPipe {
  name: IMaskerPipeName
  exec: IMaskerPipeAsync | IMaskerPipeDual
  execSync: IMaskerPipeSync | IMaskerPipeDual,
  opts?: IMaskerPipeOpts
}

True dynamic

During the execution, every pipe handler takes full control of the context. It can override next steps, change the executor impl (replace, append hook, etc), create internal masker threads, parallelize invocation queues and sync them back together, and so on.

Context

Each pipe is fed with a normalized context which consists of:

export interface IMaskerPipeInput {
  value: any                // value to process
  _value?: any              // pipe result
  id: IContextId            // ctx unique key
  context: IMaskerPipeInput // ctx self ref
  parentId?: IContextId     // parent ctx id
  registry: IMaskerRegistry       // pipe registry attached to ctx
  execute: IExecutor              // executor 
  sync: boolean                   // sync / async switch
  mode: IExecutionMode            // lagacy sync switch
  opts: IMaskerOpts               // current pipe options
  pipe?: IMaskerPipeNormalized              // current pipe ref
  pipeline: IMaskerPipelineNormalized       // actual pipeline
  originPipeline: IMaskerPipelineNormalized // origin pipeline
  [key: string]: any
}

Sync / async

Both. In different situations, each approach has pros and cons. For this reason, the masker provides a choice:

masker(data)                // async
masker.sync(data)           // sync
masker(data, {sync: true})  // sync

Documentation

• JS/TS API
• CLI

Packages

There is also a bunch of plugins, that extend the available masking scenarios. Please follow their internal docs.

Package	Description	Version
@qiwi/masker	Composite data masking utility with common pipeline preset
masquer	CLI for @qiwi/masker
@qiwi/masker-common	Masker common components: interfaces, executor, utils
@qiwi/masker-debug	Debug plugin to observe pipe effects
@qiwi/masker-infra	Infra package: build configs, tools, etc
@qiwi/masker-json	Plugin to search and parse JSONs chunks in strings
@qiwi/masker-limiter	Plugin to limit masking steps count and duration
@qiwi/masker-pan	Plugin to search and conceal PANs
@qiwi/masker-plain	Plugin to substitute any kind of data with ***
@qiwi/masker-schema	Masker schema builder and executor
@qiwi/masker-secret-key	Plugin to hide sensitive data by key/path pattern match
@qiwi/masker-secret-value	Plugin to conceal substrings by pattern match
@qiwi/masker-split	Executor hook to recursively process any object inners
@qiwi/masker-strike	Plugin to strikethough any non-space string chars
@qiwi/masker-trycatch	Executor hook to capture and handle exceptions

Contributing

Feel free to open any issues: for bugs, feature requests or questions. You're always welcome to suggest a PR. Just fork this repo, write some code, add some tests and push your changes. Any feedback is appreciated.

License

MIT