Guide to Embedding

July 24, 2020 · View on GitHub

Guide to Embedding

(in progress... TODO)

Grabbing the Correct Files

Embedding starts with dropping in the correct files:

C99

Copy the files into your project:

src/sink.h
src/sink.c

Their location doesn't matter, as long as they're in the same directory. Add sink.c to your build so it's compiled.

TypeScript

Copy the files into your project:

dist/sink.ts
dist/sink.d.ts (declaration file, optional)

Import using:

import sink = require('./sink.js');

JavaScript

Copy the file into your project:

dist/sink.js

Use in node via:

var sink = require('./sink');

Or the browser via RequireJS:

<script src="require.js"></script>
<script>
requirejs(['sink'], function(sink){
  ... your code here ...
});
</script>

API

The API is very similar across the implementations. The TypeScript API is accessed via sink.some_function, whereas the C99 API is sink_some_function. See the appropriate header files (sink.h and sink.d.ts) for the exact declarations.

The API is in four basic sections:

Section	Description
Script API	Loading a program into memory, compiling if necessary
Context API	Executing a program, pausing/resuming execution
Standard Library API	Executing commands from the standard library inside a context
Misc/Helper Functions	Assorted functions to make life easier

Script API

The Script API is used for loading a program into memory. It can load a script and compile it into bytecode, or load bytecode directly. It does not execute any code.

Functions	Description
`scr_new`	Create a new Script object
`scr_addpath`	Add a search path for including/embedding files
`scr_incbody`	Provide a string to be included for a special filename
`scr_incfile`	Provide a file to be included for a special filename
`scr_getfile`	Get the fully resolved filename of the script
`scr_getcwd`	Get the current working directory of the script
`scr_geterr`	Get any error message associated with the script
`scr_cleanup`	Add user-defined objects to be freed when Script is freed
`scr_setuser`	Set a user-defined value associated with the Script
`scr_getuser`	Get a previously set user-defined value
`scr_loadfile`	Load a file through the include system
`scr_write`	Write file contents into the Script object
`scr_level`	Get the level of nesting for REPLs
`scr_dump`	Dump the compiled bytecode
`scr_free`	Free a Script object

scr_new

Create a new Script object.

typedef enum {
  SINK_FSTYPE_NONE,
  SINK_FSTYPE_FILE,
  SINK_FSTYPE_DIR
} sink_fstype;

typedef sink_fstype (*sink_fstype_f)(sink_scr scr, const char *file, void *incuser);
typedef bool (*sink_fsread_f)(sink_scr scr, const char *file, void *incuser);

typedef struct {
  sink_fstype_f f_fstype;
  sink_fsread_f f_fsread;
  void *user;
} sink_inc_st;

sink_scr sink_scr_new(sink_inc_st inc, const char *curdir, bool posix, bool repl);

enum sink.fstype {
  NONE,
  FILE,
  DIR
}

type sink.fstype_f = (scr: sink.scr, file: string, incuser: any) => Promise<sink.fstype>;
type sink.fsread_f = (scr: sink.scr, file: string, incuser: any) => Promise<boolean>;

interface sink.inc_st {
  f_fstype: sink.fstype_f;
  f_fsread: sink.fsread_f;
  user?: any;
}

function sink.scr_new(inc: sink.inc_st, curdir: string | null, posix: boolean, repl: boolean):
  sink.scr;

`inc`

An object that provides functions for the compiler to read files from the filesystem.

When a script uses include or embed, the compiler will query these functions (using the search path) to figure out how to resolve to the correct file, and read the file.

The f_fstype function should query the filesystem, and return one of the results: NONE (i.e., SINK_FSTYPE_NONE in C, and sink.fstype.NONE in TypeScript) for a file that doesn't exist, FILE for a file, and DIR for a directory.

The f_fsread function should attempt to open the provided file, and write it to the Script object using scr_write. It should return true if the file was read successfully, and false if the file failed to be read.

See cmd.c or cmd.ts for example implementations of these functions. Note that the TypeScript/JavaScript version must return a Promise.

The user field is passed through to the incuser argument in the functions, at your discretion.

`curdir`

The current working directory (or null/NULL). This must be an absolute path.

This is used when a script includes or embeds a relative path, in order to construct an absolute path.

`posix`

If true, then path routines will use POSIX logic. It will treat / as a path seperator, and any path starting with / will be considered an absolute path.

If false, then path routines will use Windows logic. It will treat / and \ as path seperators, join paths using \, and paths starting with a drive letter or network name will be considered absolute (i.e., c:\foo or \\host\computer\foo).

`repl`

Flag indicating whether the script is a REPL.

REPL scripts have slightly different rules during compilation, centered around allowing the user to re-define symbols.

For example, the following script will result in a compile-time error, but works fine in a REPL entered one line at a time:

var test = 1
var test = 2
say test # => outputs 2 in a REPL

Normally, defining something twice will result in a compile-time error (Cannot redefine "test"). In a REPL, re-definitions are allowed.

scr_addpath

Add a path to the list of search paths during an include or embed statement.

void sink_scr_addpath(sink_scr scr, const char *path);

function sink.scr_addpath(scr: sink.scr, path: string): void;

If a file with a relative path is included or embedded, sink will iterate over the search paths. If a search path is relative, it will be combined with the current working directory, along with the target file, to create an absolute path.

It's recommended to at least add "." as a search path so that the current directory will be searched when including/embedding a file.

`scr`

The Script object.

`path`

The path to add to the list of search paths.

scr_incbody

Define a module in-memory, so that if the end-user performs an include 'name', the provided source code (body) is used as the content.

void sink_scr_incbody(sink_scr scr, const char *name, const char *body);

function sink.scr_incbody(scr: sink.scr, name: string, body: string): void;

This function is used for native libraries to provide a short name that end-users can use to include the necessary definitions.

For example, suppose you want to build a shapes library, that included commands for drawing circles and rectangles. You would do:

sink.scr_incbody(scr, 'shapes',
  'declare circle     "company.shapes.circle"     \n' +
  'declare rectangle  "company.shapes.rectangle"  \n');

Then when the end-user runs include 'shapes', instead of searching for a 'shapes' file in the filesystem, it will pull the contents directly from memory.

`src`

The Script object.

`name`

The literal name the user must type in the include statement to pull in the content.

`body`

The content of the included library.

scr_incfile

Define a module that references a file, so that if the end-user performs an include 'name', the file searched for in the include system is file.

void sink_scr_incfile(sink_scr scr, const char *name, const char *file);

function sink.scr_incfile(scr: sink.scr, name: string, file: string): void;

This function has the same basic usage as scr_incbody, except instead of providing the contents directly, the file contents are stored inside an actual file.

This is useful if a library wants to make the definitions available as an external file, and simply wants to reference the file instead of having the contents directly in memory.

`scr`

The Script object.

`name`

The literal name the user must type in the include statement to search for the file.

`file`

The file to actually search for during the include.

scr_getfile

Get the fully resolved filename of the current script.

const char *sink_scr_getfile(sink_scr scr);

function sink.scr_getfile(scr: sink.scr): string | null;

If you kick-off loading the script via an scr_loadfile call with a relative file, then this function will return the fully resolved (absolute path) of the filename that was found and loaded.

`scr`

The Script object.

scr_getcwd

Returns the current working directory, as specified earlier from the scr_new call.

const char *sink_scr_getcwd(sink_scr scr);

function sink.scr_getcwd(scr: sink.scr): string | null;

scr_geterr

Returns the current error message of the script (compile-time error, or null/NULL for no error). Only needs to be checked if scr_write returns false.

const char *sink_scr_geterr(sink_scr scr);

function sink.scr_geterr(scr: sink.scr): string | null;

`scr`

The Script object.

scr_cleanup

Provide a pointer and free function to be executed when the Script object is freed (C only).

typedef void (*sink_free_f)(void *ptr);

void sink_scr_cleanup(sink_scr scr, void *cuser, sink_free_f f_free);

This is useful for libraries that allocate their own objects when loading into a sink Script object. They can ensure the objects are cleaned up when the Script object is freed.

`scr`

The Script object.

`cuser`

The pointer to be freed when the Script object is freed.

`f_free`

The function used to free cuser.

scr_setuser

Set a user-defined value associated with the Script object. The value can be retrieved by scr_getuser.

typedef void (*sink_free_f)(void *ptr);

void sink_scr_setuser(sink_scr scr, void *user, sink_free_f f_free);

function sink.scr_setuser(scr: sink.scr, user: any): void;

`scr`

The Script object.

`user`

The user-defined value.

`f_free`

The function used to free the object (C only). If the user-defined value is overwritten by another call to scr_setuser, or if the Script object is freed, then the current user value is freed via f_free.

scr_getuser

Get the user-defined value associated with the Script object, previously set with scr_setuser.

void *sink_scr_getuser(sink_scr scr);

function sink.scr_getuser(scr: sink.scr): any;

`scr`

The Script object.

scr_loadfile

Load the script's file from the include filesystem. Returns true if the file was loaded without error, or false if there was an error. The error is accessible via scr_geterr.

bool sink_scr_loadfile(sink_scr scr, const char *file);

function sink.scr_loadfile(scr: sink.scr, file: string): Promise<boolean>;

A script can be loaded directly via scr_write, but the compiler won't have any filename information. Instead, using scr_loadfile will kick off loading the data using the include system, so that any errors are correctly identified as coming from the source file.

This will query for the file using f_fstype, and load the file using f_fsread, which should call scr_write.

Note that the TypeScript/JavaScript version returns a Promise.

`scr`

The Script object.

`file`

The file to load using the include system.

scr_write

Write the contents of a file or buffer into the Script object. This is how all data is loaded into the Script. Returns true if the data was processed without error, or false if a compile-time error occurred. The error can be retrieved via scr_geterr.

bool sink_scr_write(sink_scr scr, int size, const uint8_t *bytes);

function sink.scr_write(scr: sink.scr, bytes: string): Promise<boolean>;

Loading source code into the Script object is done with scr_write. The function can be called directly after creating a Script object, or it can be called indirectly through scr_loadfile -- which will eventually call f_fsread which should call scr_write to write the data.

Note that the TypeScript/JavaScript version returns a Promise.

`scr`

The Script object.

`size`

The size of the bytes array (C only).

`bytes`

The raw bytes to load. Note: in TypeScript/JavaScript, the string is interpretted as 'binary' encoding.

scr_level

Returns the current nesting level for a REPL. A value of 0 indicates no nesting.

int sink_scr_level(sink_scr scr);

function sink.scr_level(scr: sink.scr): number;

When typing in a series of lines in a REPL, the nesting level is how many indentations should be shown in the prompt. For example, a session might look like this:

 1: if rand.num < 0.5     # nesting level 0
 2... for var v: range 5  # nesting level 1
 3..... say v + 100       # nesting level 2
 4..... end               # nesting level 2
 5... end                 # nesting level 1
100
101
102
103
104
 6:                       # nesting level 0

The nesting level determines how many periods to draw to indent the line, and when the nesting level hits 0, it's known that a full statement has been entered, so the context is ran to catch up to what's been entered.

`scr`

The Script object.

scr_dump

Output the compiled bytecode using the supplied f_dump function.

typedef size_t (*sink_dump_f)(const void *restrict ptr, size_t size, size_t nitems,
  void *restrict dumpuser);

void sink_scr_dump(sink_scr scr, bool debug, void *user, sink_dump_f f_dump);

type dump_f = (data: string, dumpuser: any) => void;

function sink.scr_dump(scr: sink.scr, debug: boolean, user: any, f_dump: sink.dump_f): void;

`scr`

The Script object.

`debug`

Flag whether to output debug information.

Debug information allows for stacktraces to be populated for run-time errors, but increases the size of the bytecode, and exposes filenames and line numbers to the end-user.

`user`

User-supplied value that is passed directly to f_dump as parameter dumpuser.

`f_dump`

The function called to dump the bytecode.

Note that this function has the same signature as fwrite in the C implementation, so that the file pointer can be passed in as user, and fwrite can be passed in as f_dump, for convenience. The size_t return value is ignored.

The TypeScript implementation is passed string data in 'binary' encoding.

scr_free

Free the Script object (C only).

void sink_scr_free(sink_scr scr);

Note this will also free any scr_setuser value, and any scr_cleanup pointers that have been associated with the Script object, before freeing the Script object itself.

`scr`

The Script object.

Context API

The Context API is used for executing a Script. It controls how and when to run the bytecode on the virtual machine.

Functions	Description
`ctx_new`	Create a new Context object
`ctx_getstatus`	Get the status of the Context's virtual machine
`ctx_geterr`	Get any error message associated with the run-time
`ctx_native`	Add a native command using a string
`ctx_nativehash`	Add a native command using a 64-bit hash
`ctx_cleanup`	Add user-defined objects to be freed when Context is freed (C only)
`ctx_setuser`	Set a user-defined value associated with the Context
`ctx_getuser`	Get a previously set user-defined value
`ctx_addusertype`	Add a new usertype, for use with list's user data
`ctx_getuserfree`	Get a usertype's free function (C only)
`ctx_getuserhint`	Get a usertype's hint string
`ctx_settimeout`	Set a timeout so the machine pauses itself
`ctx_gettimeout`	Get the current timeout value
`ctx_consumeticks`	Decrease the current tick counter by an amount
`ctx_forcetimeout`	Force a timeout to occur immediately
`ctx_run`	Run the virtual machine
`ctx_free`	Free a Context object (C only)
`waiter`	Create a Wait object (C only)
`done`	Create a Wait object that already has a result (C only)
`then`	Attach a handler to a Wait object (C only)
`result`	Provide a result for a Wait object (C only)

ctx_new

Create a new Context object.

typedef sink_wait (*sink_io_f)(sink_ctx ctx, sink_str str, void *iouser);

typedef struct {
  sink_io_f f_say;
  sink_io_f f_warn;
  sink_io_f f_ask;
  void *user;
} sink_io_st;

sink_ctx sink_ctx_new(sink_scr scr, sink_io_st io);

type sink.io_f = (ctx: sink.ctx, str: sink.str, iouser: any) => Promise<sink.val>;

interface sink.io_st {
    f_say?: sink.io_f;
    f_warn?: sink.io_f;
    f_ask?: sink.io_f;
    user?: any;
}

function sink.ctx_new(scr: sink.scr, io: sink.io_st): sink.ctx;

`scr`

The Script object to execute in the Context.

`io`

The input/output functions for the machine.

The f_say, f_warn, and f_ask functions are called when the associated say, warn, and ask commands are executed in the script.

The C versions use the sink_wait system to deal with asynchronous operations.

The TypeScript versions must return a Promise.

It's recommended that f_say and f_warn always return NIL, but not required.

The user field is mapped to iouser, and can be used for anything.

ctx_getstatus

Get the status of the Context's virtual machine.

typedef enum {
  SINK_READY,
  SINK_WAITING,
  SINK_PASSED,
  SINK_FAILED
} sink_status;

sink_status sink_ctx_getstatus(sink_ctx ctx);

enum sink.status {
  READY,
  WAITING,
  PASSED,
  FAILED
}

function sink.ctx_getstatus(ctx: sink.ctx): sink.status;

Returns one of the following values:

READY - The virtual machine is ready for execution via ctx_run.
WAITING - The virtual machine is waiting for an asynchronous operation to finish.
PASSED - The virtual machine finished executing the script and exited successfully.
FAILED - The virtual machine finished executing the script and exited in failure.

Sink Command	Host Function
`+x`	`tonum`
`'' ~ x`	`tostr`
`&x`	`size`
`say`	`say`
`warn`	`warn`
`ask`	`ask`
`exit`	`exit`
`abort`	`abort`
`isnum`	`isnum`
`isstr`	`isstr`
`islist`	`islist`
`isnative`	`isnative` or `isnativehash`
`range`	`range`
`order`	`order`
`stacktrace`	`stacktrace`
`-x`	`num_neg`
`x + y`	`num_add`
`x - y`	`num_sub`
`x * y`	`num_mul`
`x / y`	`num_div`
`x % y`	`num_mod`
`x ^ y`	`num_pow`
`num.abs`	`num_abs`
`num.sign`	`num_sign`
`num.max`	`num_max`
`num.min`	`num_min`
`num.clamp`	`num_clamp`
`num.floor`	`num_floor`
`num.ceil`	`num_ceil`
`num.trunc`	`num_trunc`
`num.nan`	`num_nan`
`num.inf`	`num_inf`
`num.isnan`	`num_isnan`
`num.isfinite`	`num_isfinite`
`num.e`	`num_e`
`num.pi`	`num_pi`
`num.tau`	`num_tau`
`num.sin`	`num_sin`
`num.cos`	`num_cos`
`num.tan`	`num_tan`
`num.asin`	`num_asin`
`num.acos`	`num_acos`
`num.atan`	`num_atan`
`num.atan2`	`num_atan2`
`num.log`	`num_log`
`num.log2`	`num_log2`
`num.log10`	`num_log10`
`num.exp`	`num_exp`
`num.lerp`	`num_lerp`
`num.hex`	`num_hex`
`num.oct`	`num_oct`
`num.bin`	`num_bin`
`int.new`	`int_new`
`int.not`	`int_not`
`int.and`	`int_and`
`int.or`	`int_or`
`int.xor`	`int_xor`
`int.shl`	`int_shl`
`int.shr`	`int_shr`
`int.sar`	`int_sar`
`int.add`	`int_add`
`int.sub`	`int_sub`
`int.mul`	`int_mul`
`int.div`	`int_div`
`int.mod`	`int_mod`
`int.clz`	`int_clz`
`int.pop`	`int_pop`
`int.bswap`	`int_bswap`
`rand.seed`	`rand_seed`
`rand.seedauto`	`rand_seedauto`
`rand.int`	`rand_int`
`rand.num`	`rand_num`
`rand.range`	`rand_range`
`rand.getstate`	`rand_getstate`
`rand.setstate`	`rand_setstate`
`rand.pick`	`rand_pick`
`rand.shuffle`	`rand_shuffle`
`str.new`	`str_new`
`x ~ y`	`str_cat`
`x[y:z]`	`str_slice`
`x[y:z] = w`	`str_splice`
`str.split`	`str_split`
`str.replace`	`str_replace`
`str.begins`	`str_begins`
`str.ends`	`str_ends`
`str.pad`	`str_pad`
`str.find`	`str_find`
`str.rfind`	`str_rfind`
`str.lower`	`str_lower`
`str.upper`	`str_upper`
`str.trim`	`str_trim`
`str.rev`	`str_rev`
`str.list`	`str_list`
`str.byte`	`str_byte`
`str.hash`	`str_hash`
`utf8.valid`	`utf8_valid`
`utf8.list`	`utf8_list`
`utf8.str`	`utf8_str`
`struct.size`	`struct_size`
`struct.str`	`struct_str`
`struct.list`	`struct_list`
`struct.isLE`	`struct_isLE`
`list.new`	`list_new`
`x ~ y`	`list_cat`
`x[y:z]`	`list_slice`
`x[y:z] = w`	`list_splice`
`list.shift`	`list_shift`
`list.pop`	`list_pop`
`list.push`	`list_push`
`list.unshift`	`list_unshift`
`list.append`	`list_append`
`list.prepend`	`list_prepend`
`list.find`	`list_find`
`list.rfind`	`list_rfind`
`list.join`	`list_join`
`list.rev`	`list_rev`
`list.str`	`list_str`
`list.sort`	`list_sort`
`list.rsort`	`list_rsort`
`pickle.json`	`pickle_json`
`pickle.bin`	`pickle_bin`
`pickle.val`	`pickle_val`
`pickle.valid`	`pickle_valid`
`pickle.sibling`	`pickle_sibling`
`pickle.circular`	`pickle_circular`
`pickle.copy`	`pickle_copy`
`gc.getlevel`	`gc_getlevel`
`gc.setlevel`	`gc_setlevel`
`gc.run`	`gc_run`

Function/Value	Description
`NIL`	The literal `nil` value
`bool`	Convert a boolean to a sink value
`isnil`	Test if a sink value is `nil`
`isfalse`	Test if a sink value is false (`nil`)
`istrue`	Test if a sink value is true (non-`nil`)
`typeof`	Get the type of a sink value
`castnum`	Reinterpret a sink value as a number (C only)
`num`	Convert a number to a sink value (C only)
`caststr`	Reinterpret a sink value as a string (C only)
`str_newcstr`	Create sink string by copying a C string (C only)
`str_newcstrgive`	Create sink string from a C string, giving memory ownership to sink (C only)
`str_newblob`	Create sink string by copying a blob of data (C only)
`str_newblobgive`	Create sink string from a blob of data, giving memory ownership to sink (C only)
`str_newempty`	Create an empty string (C only)
`str_newformat`	Create a string from a `printf`-formatted expression (C only)
`str_hashplain`	Hash a string directly
`castlist`	Reinterpret a sink value as a list (C only)
`list_newblob`	Create sink list by copying a list of values (C only)
`list_newblobgive`	Create sink list from a list of values, giving memory ownership to sink (C only)
`list_newempty`	Create an empty list (C only)
`list_setuser`	Set the user-defined data associated with a list
`list_hasuser`	Check if a list has a certain type of user-defined data
`list_getuser`	Get the user-defined data associated with a list
`list_joinplain`	Join a list directly
`user_new`	Create a list with user-defined data attached to it
`pickle_binstr`	Pickle a sink value into binary data for marshalling
`pickle_binstrfree`	Free the results of `pickle_binstr` (C only)
`pickle_valstr`	Convert a marshalled string into a sink value
`gc_pin`	Pin a value to prevent it from being garbage-collected (C only)
`gc_unpin`	Unpin a previously pinned value, allowing it to be garbage-collected (C only)
`abortstr`	Abort with a formatted string (C only)
`arg_bool`	Convert a native argument to a boolean
`arg_num`	Convert a native argument to a number, if possible
`arg_str`	Convert a native argument to a string, if possible
`arg_list`	Convert a native argument to a list, if possible
`arg_user`	Convert a native argument to a user-defined value, if possible
`seedauto_src`	The function that provides the source random data for `rand.seedauto`
`malloc`	The function that provides all memory allocation (C only)
`realloc`	The function that provides all memory reallocation (C only)
`free`	The function that provides all memory freeing (C only)