HtmlTinkerX & PSParseHTML - HTML processing for .NET and PowerShell

August 23, 2026 · View on GitHub

HtmlTinkerX is the shared .NET engine for parsing, extracting, auditing, formatting, crawling, and rendering web content. PSParseHTML exposes the same engine as PowerShell cmdlets.

🛠️ Project Information

Dependency guardrails, including the ChartForgeX-backed screenshot image-processing path, are documented in Docs/Dependencies.md.

What it covers

HTML parsing with AngleSharp and Html Agility Pack
object-first page reading with headings, paragraphs, tables, links, resources, and inferred repeated collections
tables, lists, forms, metadata, JSON-LD, microdata, Open Graph, application state, tokens, image candidates, and API endpoint extraction
static and rendered document audits for duplicate IDs, document metadata, accessible names, unsafe URL schemes, and heading order
bounded website crawling to offline HTML, text, Markdown, JSON, JSONL, graph, and asset datasets
Playwright sessions, interaction, screenshots, PDFs, HAR files, traces, browser recipes, cookies, storage, and SSO handoff inspection
HTML, CSS, JavaScript, and email formatting or optimization
.NET Framework 4.7.2, .NET 8, and .NET 10

PowerShell and C# entry points

The PowerShell commands are thin surfaces over HtmlTinkerX. The generated command reference covers every cmdlet.

Task	PowerShell	C# owner
Read a page as objects without selectors	`Get-HtmlPage`	`HtmlPageReader.Read`
Parse a document	`ConvertFrom-Html`	`HtmlParser.ParseWithAngleSharp`, `HtmlParser.ParseWithHtmlAgilityPack`
Extract tables	`ConvertFrom-HtmlTable`	`HtmlParser.ParseTablesWithAngleSharpDetailed`, `HtmlParser.ParseTablesWithHtmlAgilityPackDetailed`
Extract lists	`ConvertFrom-HtmlList`	`HtmlParser.ParseListsWithAngleSharpDetailed`, `HtmlParser.ParseListsWithHtmlAgilityPackDetailed`
Extract forms	`ConvertFrom-HtmlForm`	`HtmlParser.ParseFormsWithAngleSharp`
Extract metadata	`ConvertFrom-HtmlMeta`	`HtmlParser.ParseMetaTags`
Extract Open Graph data	`ConvertFrom-HtmlOpenGraph`	`HtmlParser.ParseOpenGraph`
Extract microdata	`ConvertFrom-HtmlMicrodata`	`HtmlParser.ParseMicrodataItems`
Normalize mixed page data	`Select-HtmlData`	`HtmlParsingToolbox.SelectData`
Build a page workbench and audit	`Invoke-HtmlPageWorkbench`	`HtmlPageWorkbench.AnalyzeAsync`, `HtmlDocumentAudit.Analyze`
Find interaction surfaces	`Find-HtmlInteractionSurface`	`HtmlParsingToolbox.FindInteractionSurfaceAsync`
Discover API endpoints	`Find-HtmlApiEndpoint`	`HtmlApiEndpointInventory.Build`
Compare static and rendered HTML	`Compare-HtmlStaticRendered`	`HtmlParsingToolbox.CompareStaticRendered`
Crawl and export a dataset	`Invoke-HtmlCrawl`	`HtmlCrawler.CrawlAsync`
Open or navigate a browser	`Start-HtmlBrowserSession`, `Invoke-HtmlBrowserNavigation`	`HtmlBrowser.OpenSessionAsync`, `HtmlBrowser.NavigateAsync`
Click or fill an element	`Invoke-HtmlBrowserClick`, `Set-HtmlBrowserInput`	`HtmlBrowser.ClickSelectorAsync`, `HtmlBrowser.FillInputAsync`
Capture a screenshot or PDF	`Save-HtmlBrowserScreenshot`, `Save-HtmlBrowserPdf`	`HtmlBrowser.CaptureScreenshotAsync`, `HtmlBrowser.SavePagePdfAsync`
Export HAR or trace data	`Export-HtmlBrowserHar`, `Start-HtmlBrowserTracing`, `Stop-HtmlBrowserTracing`	`HtmlBrowser.ExportHarAsync`, `HtmlBrowser.StartTracingAsync`, `HtmlBrowser.StopTracingAsync`
Test a rendered page	`Test-HtmlBrowser`	`HtmlBrowserTester`
Inline email CSS	`Optimize-Email`	`PreMailerClient.MoveCssInline`, `PreMailerClient.MoveCssInlineAsync`
Format or minify resources	`Format-Html`, `Format-Css`, `Format-JavaScript`, `Optimize-*`	`HtmlFormatter`, `HtmlOptimizer`

📦 Installation & Packages

📦 NuGet Package (C#/.NET)

dotnet add package HtmlTinkerX

🔧 PowerShell Module

Install-Module -Name PSParseHTML -AllowClobber -Force

These commands install the current stable releases. Upstream dependency version labels do not change HtmlTinkerX or PSParseHTML release channels.

📋 Package Information

📦 NuGet Package: HtmlTinkerX - Core .NET library
🔧 PowerShell Module: PSParseHTML - PowerShell cmdlets wrapper
🎯 Target Frameworks: .NET Framework 4.7.2, .NET 8.0, and .NET 10.0
💻 PowerShell Compatibility: Windows PowerShell 5.1 and PowerShell 7.4+

🚀 Quick Start

Read a page as objects

Start here when you want the content rather than the DOM:

$page = Get-HtmlPage -Url 'https://example.org/catalog'

$page.Headings
$page.Paragraphs
$page.Tables
$page.Links

# Repeated cards, rows, or listings are inferred for you.
$page.Collections | Select-Object Name, Count, Confidence
$page.Collections[0].Items

Collection items expose inferred fields as ordinary PowerShell properties, such as Title, Price, ProductLink, or Image. Field names depend on the page. Use $page.Collections[0].Fields to inspect what was recognized. The retained Selector property is provenance for troubleshooting and reusable recipes; it is not required to read the page.

The task-oriented object workflow guide shows how to inspect the result, work with articles and typed tables, choose a repeated collection by its fields, handle rendered pages, and move to explicit selectors only when you need a stable extraction recipe.

Markdown remains available as a projection:

$page.Markdown

For client-rendered pages, pass a snapshot captured by Invoke-HtmlRendering:

$snapshot = Invoke-HtmlRendering -Url $url -Snapshot
$page = Get-HtmlPage -RenderedSnapshot $snapshot

C# Example

using HtmlTinkerX;

// Read semantic page objects and inferred collections
string html = await File.ReadAllTextAsync("page.html");
HtmlPageDocument page = HtmlPageReader.Read(
    html,
    new HtmlPageReaderOptions {
        BaseUri = new Uri("https://example.org/catalog")
    });

Console.WriteLine(page.Headings[0].Text);
Console.WriteLine(page.Tables.Count);
Console.WriteLine(page.Collections[0].Items[0]["Title"]);

// Audit or use lower-level parsers when you need them
HtmlDocumentAuditResult audit = HtmlDocumentAudit.Analyze(html);

// Format and optimize resources
string formatted = HtmlFormatter.FormatHtml(html);
string minified = HtmlOptimizer.OptimizeHtml(html, cssDecodeEscapes: false);

// Browser automation
await using var session = await HtmlBrowser.OpenSessionAsync("https://example.com");
await HtmlBrowser.CaptureScreenshotAsync(session.Page, "screenshot.png");

// Offline crawl
var crawl = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    MaxDepth = 1,
    MaxPages = 10,
    UseSitemaps = true,
    RespectRobotsTxt = true,
    DeduplicatePages = true,
    OutputPath = "crawl-output"
});

Console.WriteLine(crawl.Summary.ToReportText(crawl.SitemapUrls));
Console.WriteLine(crawl.PagesCsvPath);

// AI-ready offline dataset with Markdown alongside HTML/text
var aiReady = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "main",
    IncludeMarkdown = true,
    OutputPath = "crawl-output"
});
Console.WriteLine(aiReady.Pages[0].MarkdownPath);

// Self-contained JSON document export for downstream automation
var structured = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "main",
    IncludeStructuredJson = true,
    OutputPath = "crawl-output"
});
Console.WriteLine(structured.StructuredJsonPagesJsonlPath);
Console.WriteLine(structured.OpenApiLikePath);
Console.WriteLine(structured.OpenApiPath);
Console.WriteLine(structured.OpenApiDocument["openapi"]);
Console.WriteLine(structured.OpenApiLike.StrictOpenApiEligibleOperationCount);
Console.WriteLine(structured.OpenApiLike.StrictOpenApiSkippedOperationCount);
Console.WriteLine(structured.OpenApiDocument.ContainsKey("x-htmltinkerx-promotion"));
Console.WriteLine(structured.Pages[0].StructuredJson!.Document.Summary);
Console.WriteLine(structured.Pages[0].StructuredJson!.Document.Markdown);
Console.WriteLine(structured.Pages[0].StructuredJson!.Metadata.Description);
Console.WriteLine(structured.Pages[0].StructuredJson!.Layout.NavigationCount);
Console.WriteLine(structured.Pages[0].StructuredJson!.CodeBlocks.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.CodeSamples.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.ApiEndpoints.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.Breadcrumbs.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.FaqItems.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.SpecTables.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.Callouts.Count);
Console.WriteLine(structured.Pages[0].StructuredJson!.PrimaryActions.Count);

// Built-in preset fields for docs/article/product pages, with auto mode available too
var docsJson = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "main",
    IncludeStructuredJson = true,
    StructuredJsonPreset = HtmlCrawlStructuredJsonPreset.Docs,
    OutputPath = "crawl-output"
});
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ResolvedPreset);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["mainHeading"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["navigationLinks"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["codeSamples"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["apiEndpoints"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["apiCatalog"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["apiTags"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["apiResources"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["operationIds"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["openApiLike"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["openApiPaths"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["openApiServers"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["authenticationSchemes"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["rateLimitHeaders"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["operationId"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["resource"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["tags"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["requestExamples"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["requestHeaders"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["responseHeaders"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["errorResponses"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["errorCatalog"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["successResponseSchema"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["errorResponseSchema"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["requestBodyFields"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["successResponseFields"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["errorResponseFields"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].Parameters.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].OperationId);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].Resource);
Console.WriteLine(string.Join(", ", docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].Tags));
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].BodyParameters.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].HeaderParameters.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RequestBodySchema["name"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RequestBodyFields[0].Path);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RequestBodyFields[0].Provenance[0].Kind);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RequestBodyFields[0].ConfidenceScore);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].BodyParameters[0].Format);
Console.WriteLine(string.Join(", ", docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].BodyParameters[0].EnumValues));
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].BodyParameters[0].ExampleValue);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].Authentication.Required);
Console.WriteLine(string.Join(", ", docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].Authentication.Schemes));
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RateLimit.Limit);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RequestExamples.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].RequestHeaders.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].ResponseHeaders.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].ErrorResponses.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].ErrorCatalog.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].SuccessResponseSchema["id"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].ErrorResponseSchema["error"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].SuccessResponseFields[0].Path);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].SuccessResponseFields[0].Provenance[0].Kind);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].SuccessResponseFields[0].ConfidenceScore);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].ErrorResponseFields[0].Path);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].SuccessResponseFields[0].Kind);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].SuccessResponseFields[0].ChildPaths.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiEndpoints[0].ResponseExamples.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.ApiCatalog.OperationCount);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.OpenApiLike.Paths["/v1/widgets"].Operations["post"].OperationId);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].OperationId);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].AuthenticationRef);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].ParametersRef);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].RequestHeadersRef);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].ResponseExamplesRef);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].StrictOpenApiScore);
Console.WriteLine(docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].StrictOpenApiEligible);
Console.WriteLine(string.Join(", ", docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].Provenance.PageUrls));
Console.WriteLine(string.Join(", ", docsJson.OpenApiLike.Paths["/v1/widgets"].Operations["post"].Provenance.SourceKinds));
Console.WriteLine(docsJson.OpenApiLike.Components.AuthProfiles.Count);
Console.WriteLine(docsJson.OpenApiLike.Components.FieldSets.Count);
Console.WriteLine(docsJson.OpenApiLike.Components.ParameterSets.Count);
Console.WriteLine(docsJson.OpenApiLike.Components.RequestHeaderSets.Count);
Console.WriteLine(docsJson.OpenApiLike.Components.ResponseExampleSets.Count);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["breadcrumbs"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["faqItems"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["callouts"]);
Console.WriteLine(docsJson.Pages[0].StructuredJson!.Extracted["primaryActions"]);

// Caller-defined JSON fields layered on top of the built-in structured model
var extracted = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "main",
    StructuredJsonPreset = HtmlCrawlStructuredJsonPreset.Docs,
    StructuredJsonSchema = """
    {
      "title": "Metadata.Title",
      "description": "Metadata.Description",
      "navLinks": { "selector": "nav a", "source": "page", "mode": "text", "all": true },
      "mainHeading": { "selector": "h1", "source": "selected", "mode": "text" }
    }
    """
});
Console.WriteLine(extracted.Pages[0].StructuredJson!.Extracted["title"]);
Console.WriteLine(extracted.Pages[0].StructuredJson!.Extracted["mainHeading"]);

// One-call dataset mode turns on Markdown + structured JSON defaults
var dataset = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Scenario = HtmlCrawlScenario.Dataset,
    OutputPath = "crawl-output"
});
Console.WriteLine(dataset.Pages[0].MarkdownPath);
Console.WriteLine(dataset.Pages[0].StructuredJsonPath);
Console.WriteLine(dataset.Pages[0].StructuredJson!.ResolvedPreset);

// Keep full tracking query strings when a site uses them as real page identity
var exactUrls = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    IgnoreTrackingQueryParameters = false
});

// Allow non-HTML responses when you intentionally want them in the crawl
var permissive = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    SkipKnownAssetUrls = false,
    RestrictToAllowedContentTypes = false
});

// Download discovered images/documents into the offline dataset
var richSnapshot = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    MaxDepth = 1,
    DownloadAssets = true,
    OutputPath = "crawl-output"
});
// Stored HTML now points at local ../assets/... paths by default

// Hybrid crawl for JavaScript-heavy pages with cleanup of known noisy blocks
var hybrid = await HtmlCrawler.CrawlAsync("https://example.com/app", new HtmlCrawlOptions {
    AutoRender = true,
    WaitForSelector = "#main",
    DismissSelectors = { ".cookie-banner button", "#consent-accept" },
    DismissTexts = { "Accept", "I agree" },
    ClickSelectors = { ".load-more", ".expand-details" },
    ClickTexts = { "Load more", "Show more" },
    InteractionRepeatCount = 2,
    AutoScroll = true,
    ExcludeSelectors = { ".language-switcher", ".share-links", ".related-posts" }
});
Console.WriteLine($"{hybrid.Pages[0].RenderMode}: {hybrid.Pages[0].RenderReason}");
Console.WriteLine(string.Join(", ", hybrid.Pages[0].AppliedInteractions));
Console.WriteLine(hybrid.Summary.ToReportText(hybrid.SitemapUrls));

// Choose how content is selected before cleanup
var raw = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "#main",
    ContentMode = HtmlCrawlContentMode.Raw
});
var focused = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "main",
    ContentMode = HtmlCrawlContentMode.Focused
});
var reader = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    ContentMode = HtmlCrawlContentMode.Reader,
    CompareContentModes = true,
    ReaderMinimumWordCount = 30,
    ReaderMinimumScore = 40
});
Console.WriteLine($"{reader.Pages[0].ContentModeUsed} / {reader.Pages[0].ContentSelectionReasonCode}");
Console.WriteLine(reader.Pages[0].ContentElementSelectorHint);
Console.WriteLine($"{reader.Pages[0].ContentSelectionScore} across {reader.Pages[0].ReaderCandidateCount} reader candidates");
Console.WriteLine(string.Join(", ", reader.Pages[0].ContentComparisons.Select(c => $"{c.Mode}:{c.WordCount}")));
Console.WriteLine(reader.Pages[0].ContentComparisonDeltaSummary);
Console.WriteLine(reader.Pages[0].ContentComparisonPreviewSummary);
Console.WriteLine(reader.Summary.ContentComparisonWinnerPreviewSamples["Reader"]);

// Remove noisy blocks by class or id while keeping smart cleanup enabled
var clean = await HtmlCrawler.CrawlAsync("https://example.com/docs", new HtmlCrawlOptions {
    Selector = "main",
    ExcludeClasses = { "promo-box", "doc-tools" },
    ExcludeIds = { "reader-tools" }
});

// Start from an intent-focused scenario instead of hand-tuning many knobs
var docsScenario = await HtmlCrawler.CrawlAsync("https://docs.example.com/", new HtmlCrawlOptions {
    Scenario = HtmlCrawlScenario.Docs
});
Console.WriteLine(docsScenario.AppliedScenario);
Console.WriteLine(docsScenario.Pages[0].ContentModeUsed);

// Reuse a built-in profile for a common site family
var profiled = await HtmlCrawler.CrawlAsync("https://docs.example.com/", new HtmlCrawlOptions {
    ProfileName = "docs-content"
});
Console.WriteLine(profiled.AppliedProfileName);
Console.WriteLine(profiled.AppliedProfileReasonCode);
Console.WriteLine(profiled.Pages[0].ContentComparisonDeltaSummary);

PowerShell Example

# Parse HTML tables from a webpage
$tables = ConvertFrom-HtmlTable -Url 'https://example.com'

# Format and optimize resources
$formatted = Format-HTML -Path 'page.html'
$minified = Optimize-HTML -Path 'page.html'

# Browser automation
$session = Start-HtmlBrowserSession -Url 'https://example.com'
Save-HtmlBrowserScreenshot -Session $session -OutFile 'screenshot.png'
Close-HtmlBrowserSession -Session $session

# Offline crawl
$crawl = Invoke-HTMLCrawl -Url 'https://example.com/docs' -MaxDepth 1 -MaxPages 10 -DeduplicatePages
$crawl.Pages | Select-Object Url, Title, Depth
$crawl.SkippedPages | Select-Object Url, SkipReason
$crawl.Summary.ToReportText($crawl.SitemapUrls)

# AI-ready offline dataset with Markdown alongside HTML/text
$aiReady = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector 'main' -IncludeMarkdown -OutPath '.\crawl-output'
$aiReady.Pages | Select-Object Url, MarkdownPath

# Self-contained JSON document export for downstream automation
$structured = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector 'main' -IncludeStructuredJson -OutPath '.\crawl-output'
$structured.StructuredJsonPagesJsonlPath
$structured.OpenApiLikePath
$structured.OpenApiPath
$structured.OpenApiDocument["openapi"]
$structured.OpenApiLike.StrictOpenApiEligibleOperationCount
$structured.OpenApiLike.StrictOpenApiSkippedOperationCount
$structured.OpenApiDocument.ContainsKey("x-htmltinkerx-promotion")
$structured.Pages | Select-Object Url, StructuredJsonPath, @{ N = 'Summary'; E = { $_.StructuredJson.Document.Summary } }
$structured.Pages | Select-Object Url, @{ N = 'Description'; E = { $_.StructuredJson.Metadata.Description } }, @{ N = 'NavigationCount'; E = { $_.StructuredJson.Layout.NavigationCount } }
$structured.Pages | Select-Object Url, @{ N = 'CodeBlocks'; E = { $_.StructuredJson.CodeBlocks.Count } }, @{ N = 'Breadcrumbs'; E = { $_.StructuredJson.Breadcrumbs.Count } }, @{ N = 'FaqItems'; E = { $_.StructuredJson.FaqItems.Count } }
$structured.Pages | Select-Object Url, @{ N = 'CodeSamples'; E = { $_.StructuredJson.CodeSamples.Count } }, @{ N = 'ApiEndpoints'; E = { $_.StructuredJson.ApiEndpoints.Count } }
$structured.Pages | Select-Object Url, @{ N = 'AuthenticatedEndpoints'; E = { $_.StructuredJson.ApiEndpoints.Where({ $_.Authentication.Required -ne $null -or $_.Authentication.Schemes.Count -gt 0 -or $_.Authentication.Headers.Count -gt 0 }).Count } }, @{ N = 'RateLimitedEndpoints'; E = { $_.StructuredJson.ApiEndpoints.Where({ $_.RateLimit.Mentioned -or $_.RateLimit.StatusCode -ne $null }).Count } }
$structured.Pages | Select-Object Url, @{ N = 'ApiErrorResponses'; E = { $_.StructuredJson.ApiEndpoints.ForEach({ $_.ErrorResponses.Count }) | Measure-Object -Sum | Select-Object -ExpandProperty Sum } }
$structured.Pages | Select-Object Url, @{ N = 'SpecTables'; E = { $_.StructuredJson.SpecTables.Count } }, @{ N = 'Callouts'; E = { $_.StructuredJson.Callouts.Count } }, @{ N = 'PrimaryActions'; E = { $_.StructuredJson.PrimaryActions.Count } }

# Built-in preset fields for common page types
$docs = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector 'main' -IncludeStructuredJson -StructuredJsonPreset Docs -OutPath '.\crawl-output'
$docs.Pages | Select-Object Url, @{ N = 'Preset'; E = { $_.StructuredJson.ResolvedPreset } }, @{ N = 'MainHeading'; E = { $_.StructuredJson.Extracted["mainHeading"] } }, @{ N = 'ApiEndpoints'; E = { $_.StructuredJson.Extracted["apiEndpoints"] } }, @{ N = 'Breadcrumbs'; E = { $_.StructuredJson.Extracted["breadcrumbs"] } }, @{ N = 'PrimaryActions'; E = { $_.StructuredJson.Extracted["primaryActions"] } }
$docs.Pages | Select-Object Url, @{ N = 'ApiCatalog'; E = { $_.StructuredJson.Extracted["apiCatalog"] } }, @{ N = 'ApiTags'; E = { $_.StructuredJson.Extracted["apiTags"] } }, @{ N = 'ApiResources'; E = { $_.StructuredJson.Extracted["apiResources"] } }, @{ N = 'OperationIds'; E = { $_.StructuredJson.Extracted["operationIds"] } }
$docs.Pages | Select-Object Url, @{ N = 'OpenApiLike'; E = { $_.StructuredJson.Extracted["openApiLike"] } }, @{ N = 'OpenApiPaths'; E = { $_.StructuredJson.Extracted["openApiPaths"] } }, @{ N = 'OpenApiServers'; E = { $_.StructuredJson.Extracted["openApiServers"] } }
$docs.Pages | Select-Object Url, @{ N = 'ApiParameterCount'; E = { $_.StructuredJson.ApiEndpoints[0].Parameters.Count } }, @{ N = 'OperationId'; E = { $_.StructuredJson.ApiEndpoints[0].OperationId } }, @{ N = 'Resource'; E = { $_.StructuredJson.ApiEndpoints[0].Resource } }, @{ N = 'Tags'; E = { $_.StructuredJson.ApiEndpoints[0].Tags -join ', ' } }, @{ N = 'ResponseExampleCount'; E = { $_.StructuredJson.ApiEndpoints[0].ResponseExamples.Count } }
$docs.OpenApiLike.Paths['/v1/widgets'].Operations['post']
$docs.OpenApiLike.Paths['/v1/widgets'].Operations['post'].StrictOpenApiScore
$docs.OpenApiLike.Paths['/v1/widgets'].Operations['post'].StrictOpenApiEligible
$docs.OpenApiLike.Paths['/v1/widgets'].Operations['post'].Provenance.PageUrls
$docs.OpenApiLike.Paths['/v1/widgets'].Operations['post'].Provenance.SourceKinds
$docs.OpenApiLike.Components.AuthProfiles
$docs.OpenApiLike.Components.FieldSets
$docs.OpenApiLike.Components.ParameterSets
$docs.OpenApiLike.Components.RequestHeaderSets
$docs.OpenApiLike.Components.ResponseExampleSets
$docs.Pages | Select-Object Url, @{ N = 'BodyParameterCount'; E = { $_.StructuredJson.ApiEndpoints[0].BodyParameters.Count } }, @{ N = 'HeaderParameterCount'; E = { $_.StructuredJson.ApiEndpoints[0].HeaderParameters.Count } }, @{ N = 'BodySchemaNameType'; E = { $_.StructuredJson.ApiEndpoints[0].RequestBodySchema["name"] } }
$docs.Pages | Select-Object Url, @{ N = 'BodyParameterFormat'; E = { $_.StructuredJson.ApiEndpoints[0].BodyParameters[0].Format } }, @{ N = 'BodyParameterEnum'; E = { $_.StructuredJson.ApiEndpoints[0].BodyParameters[0].EnumValues -join ', ' } }, @{ N = 'BodyParameterExample'; E = { $_.StructuredJson.ApiEndpoints[0].BodyParameters[0].ExampleValue } }
$docs.Pages | Select-Object Url, @{ N = 'RequestFieldSource'; E = { $_.StructuredJson.ApiEndpoints[0].RequestBodyFields[0].Provenance[0].Kind } }, @{ N = 'SuccessFieldSource'; E = { $_.StructuredJson.ApiEndpoints[0].SuccessResponseFields[0].Provenance[0].Kind } }
$docs.Pages | Select-Object Url, @{ N = 'RequestFieldConfidence'; E = { $_.StructuredJson.ApiEndpoints[0].RequestBodyFields[0].ConfidenceScore } }, @{ N = 'SuccessFieldConfidence'; E = { $_.StructuredJson.ApiEndpoints[0].SuccessResponseFields[0].ConfidenceScore } }
$docs.Pages | Select-Object Url, @{ N = 'RequestBodyFields'; E = { $_.StructuredJson.Extracted["requestBodyFields"] } }, @{ N = 'SuccessResponseFields'; E = { $_.StructuredJson.Extracted["successResponseFields"] } }, @{ N = 'ErrorResponseFields'; E = { $_.StructuredJson.Extracted["errorResponseFields"] } }
$docs.Pages | Select-Object Url, @{ N = 'SuccessFieldKind'; E = { $_.StructuredJson.ApiEndpoints[0].SuccessResponseFields[0].Kind } }, @{ N = 'SuccessFieldChildren'; E = { $_.StructuredJson.ApiEndpoints[0].SuccessResponseFields[0].ChildPaths -join ', ' } }
$docs.Pages | Select-Object Url, @{ N = 'AuthRequired'; E = { $_.StructuredJson.ApiEndpoints[0].Authentication.Required } }, @{ N = 'AuthSchemes'; E = { $_.StructuredJson.Extracted["authenticationSchemes"] } }, @{ N = 'RateLimit'; E = { $_.StructuredJson.ApiEndpoints[0].RateLimit.Limit } }, @{ N = 'RateLimitHeaders'; E = { $_.StructuredJson.Extracted["rateLimitHeaders"] } }
$docs.Pages | Select-Object Url, @{ N = 'RequestExamples'; E = { $_.StructuredJson.Extracted["requestExamples"] } }, @{ N = 'RequestHeaders'; E = { $_.StructuredJson.Extracted["requestHeaders"] } }, @{ N = 'RequestExampleCount'; E = { $_.StructuredJson.Extracted["requestExampleCount"] } }
$docs.Pages | Select-Object Url, @{ N = 'ResponseHeaders'; E = { $_.StructuredJson.Extracted["responseHeaders"] } }, @{ N = 'ErrorResponses'; E = { $_.StructuredJson.Extracted["errorResponses"] } }, @{ N = 'ErrorResponseCount'; E = { $_.StructuredJson.Extracted["errorResponseCount"] } }, @{ N = 'ErrorCatalog'; E = { $_.StructuredJson.Extracted["errorCatalog"] } }
$docs.Pages | Select-Object Url, @{ N = 'SuccessResponseSchema'; E = { $_.StructuredJson.Extracted["successResponseSchema"] } }, @{ N = 'ErrorResponseSchema'; E = { $_.StructuredJson.Extracted["errorResponseSchema"] } }

# Caller-defined JSON fields layered on top of the built-in structured model
$schema = @'
{
  "title": "Metadata.Title",
  "description": "Metadata.Description",
  "navLinks": { "selector": "nav a", "source": "page", "mode": "text", "all": true },
  "mainHeading": { "selector": "h1", "source": "selected", "mode": "text" }
}
'@
$extracted = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector 'main' -StructuredJsonPreset Docs -StructuredJsonSchema $schema
$extracted.Pages | Select-Object Url, @{ N = 'Title'; E = { $_.StructuredJson.Extracted["title"] } }, @{ N = 'MainHeading'; E = { $_.StructuredJson.Extracted["mainHeading"] } }

# One-call dataset mode turns on Markdown + structured JSON defaults
$dataset = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Scenario Dataset -OutPath '.\crawl-output'
$dataset.Pages | Select-Object Url, MarkdownPath, StructuredJsonPath, @{ N = 'Preset'; E = { $_.StructuredJson.ResolvedPreset } }

# Preserve full tracked URLs when query strings are meaningful
$exactUrls = Invoke-HTMLCrawl -Url 'https://example.com/docs' -KeepTrackingQueryParameters

# Allow non-HTML responses such as PDFs when needed
$permissive = Invoke-HTMLCrawl -Url 'https://example.com/docs' -AllowAssetUrls -AllowAnyContentType

# Download discovered images/documents into the offline dataset
$richSnapshot = Invoke-HTMLCrawl -Url 'https://example.com/docs' -MaxDepth 1 -DownloadAssets -OutPath '.\crawl-output'
# Stored HTML now points at local ../assets/... paths by default
# Internal page links are also rewritten to local saved .html files by default

# Hybrid crawl for JavaScript-heavy pages with lazy loading and noisy chrome removal
$hybrid = Invoke-HTMLCrawl -Url 'https://example.com/app' -AutoRender -WaitForSelector '#main' -DismissSelector '.cookie-banner button', '#consent-accept' -DismissText 'Accept', 'I agree' -ClickSelector '.load-more', '.expand-details' -ClickText 'Load more', 'Show more' -InteractionRepeatCount 2 -AutoScroll -ExcludeSelector '.language-switcher', '.share-links', '.related-posts'
$hybrid.Pages | Select-Object Url, RenderMode, RenderReason, AppliedInteractions
$hybrid.Summary.ToReportText($hybrid.SitemapUrls)
# summary now includes interaction totals and per-interaction counts

# Remove noisy blocks by class or id, and keep smart cleanup enabled by default
$clean = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector 'main' -ExcludeClass 'promo-box', 'doc-tools' -ExcludeId 'reader-tools'
# Use -DisableSmartContentCleanup if you want the raw selected content without heuristic pruning

# Start from an intent-focused scenario instead of hand-tuning many knobs
$docsScenario = Invoke-HTMLCrawl -Url 'https://docs.example.com/' -Scenario Docs
$docsScenario.AppliedScenario
$docsScenario.Pages[0] | Select-Object ContentModeUsed, ContentSelectionReasonCode

# Choose how content is selected before cleanup
$raw = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector '#main' -ContentMode Raw
$focused = Invoke-HTMLCrawl -Url 'https://example.com/docs' -Selector 'main' -ContentMode Focused
$reader = Invoke-HTMLCrawl -Url 'https://example.com/docs' -ContentMode Reader -CompareContentModes -ReaderMinimumWordCount 30 -ReaderMinimumScore 40
$reader.Pages | Select-Object Url, ContentModeUsed, ContentSelectionReasonCode, ContentElementSelectorHint, ContentSelectionScore, ReaderCandidateCount, ReaderRootElementSelectorHint
$reader.Pages[0].ContentComparisons | Select-Object Mode, ReasonCode, ElementSelectorHint, WordCount, Summary
$reader.Pages[0] | Select-Object BestContentComparisonMode, BestContentComparisonReasonCode, BestContentComparisonWordCount, RunnerUpContentComparisonMode, BestContentComparisonWordDelta, ContentComparisonDeltaSummary, ContentComparisonPreviewSummary
$reader.Summary.ContentComparisonWinnerPreviewSamples
# persisted index.html now shows the same compact deltas, for example:
# Reader 0 | Focused -12 | Raw -37
# and a side-by-side preview line, for example:
# Reader 142w @ article: Hello main article... | Focused 130w @ main: Hello main... | Raw 118w: Menu item Hello...
# summary/report output also includes one representative preview sample per winning mode

# Reuse a built-in profile for a common site family
$profiled = Invoke-HTMLCrawl -Url 'https://docs.example.com/' -Profile 'docs-content'
$profiled.AppliedProfileName
$profiled.AppliedProfileReasonCode
$profiled.Pages[0] | Select-Object ContentModeUsed, BestContentComparisonMode, ContentComparisonDeltaSummary
# Available profile names: api-docs-content, docs-content, wordpress-content
# Unknown names fail fast and report the built-in values
# docs-content also defaults to Reader mode and enables comparison mode so tuning output is available immediately

# AutoProfile can also infer the generic WordPress profile from page markers
$wordpress = Invoke-HTMLCrawl -Url 'https://example-blog.com/' -AutoProfile
$wordpress.AppliedProfileName
$wordpress.AppliedProfileReasonCode

# AutoProfile can also infer a documentation-style profile from docs markers
$docs = Invoke-HTMLCrawl -Url 'https://docs.example.com/' -AutoProfile
$docs.AppliedProfileName
$docs.AppliedProfileReasonCode

# AutoProfile can also infer API documentation profiles from Swagger/ReDoc-style markers
$apiDocs = Invoke-HTMLCrawl -Url 'https://api.example.com/docs/' -AutoProfile
$apiDocs.AppliedProfileName
$apiDocs.AppliedProfileReasonCode

# Load custom profiles from JSON
$custom = Invoke-HTMLCrawl -Url 'https://docs.example.com/' -Profile 'custom-docs' -ProfilePath '.\crawl-profiles.json'
$custom.AppliedProfileName
$custom.AppliedProfileReasonCode

# Example custom profile file snippet
# [
#   {
#     "name": "custom-docs",
#     "hosts": [ "docs.example.com" ],
#     "selector": "article",
#     "contentMode": "Reader",
#     "readerMinimumWordCount": 30,
#     "readerMinimumScore": 40,
#     "excludeClasses": [ "sidebar", "feedback-box" ]
#   }
# ]

# Inspect built-in or custom profiles
Get-HtmlCrawlProfile
Get-HtmlCrawlProfile -Path '.\crawl-profiles.json' -Name 'custom-docs'

# Resume a previous crawl snapshot
$resumed = Invoke-HTMLCrawl -Url 'https://example.com/docs' -ResumePath '.\crawl-output' -OutPath '.\crawl-output'

Persisted crawl artifacts now include:

crawl-result.json for the manifest and resume state
index.html as a browsable offline entry point into the saved dataset
pages/ with per-page .html, .txt, optional .md, and .json sidecar manifests, including extraction metadata such as content mode, selection reason, and selected element hints
pages.jsonl and pages.csv for page-level datasets
skipped-pages.jsonl for skipped content-page candidates
skipped-assets.jsonl for discovered asset/document URLs that were intentionally not crawled as pages
links.jsonl for discovered page-to-page links
assets/ and assets.jsonl for downloaded images/documents when DownloadAssets is enabled
chunks.jsonl for deduplicated text chunks ready for local search/RAG pipelines
graph.json for page nodes and cross-page link edges with offline degree metadata
summary.json and summary.txt for machine-readable and human-readable crawl reports

The crawl reports and per-page manifests now also expose extraction observability data, so you can tell whether a page used Raw, Focused, or Reader mode, whether it matched an exact selector or fell back to semantic/full-document selection, which element was ultimately used, and in reader mode what score/candidate count led to that decision.

If you want a simpler product-style starting point, use Scenario in C# or -Scenario in PowerShell. Scenarios apply high-level defaults first, then built-in profiles, auto-profile detection, and explicit options can refine them. For example:

Content prefers clean readable extraction and canonical/deduplicated pages.
Archive favors offline browsing by turning on asset download and disabling aggressive cleanup.
Docs applies article-first documentation defaults and docs-chrome cleanup.
Dataset enables reader-style extraction plus comparison diagnostics and deduplication for downstream pipelines.

If you enable CompareContentModes in C# or -CompareContentModes in PowerShell, each page also gets a compact side-by-side comparison for Raw, Focused, and Reader extraction, plus a computed best mode winner based on extracted text size with a slight preference for cleaner modes when the results are close. The persisted manifests and offline index now also show the runner-up mode and the word-count delta between them, which makes it much easier to see whether the winning mode was a clear improvement or only a marginal cleanup win.

Profiles can also carry tuning defaults. The built-in docs-content and api-docs-content profiles default to Reader mode and enable comparison mode automatically, with tuned reader thresholds for their content shapes. Site-specific tuning should live in custom profile JSON loaded through ProfilePath / -ProfilePath, which can opt in with "contentMode": "Reader", "readerMinimumWordCount": 30, "readerMinimumScore": 40, and "compareContentModes": true.

The crawl result, summary, page manifests, pages.jsonl/csv, and offline index.html now also expose why a profile was chosen through AppliedProfileReasonCode and AppliedProfileReason, so it is easy to tell explicit selection from host matching, WordPress markers, docs markers, and API-doc markers.

By default the crawler also normalizes away common tracking query parameters such as utm_*, fbclid, and gclid so the dataset does not fill up with duplicate tracked URLs. Use IgnoreTrackingQueryParameters = false in C# or -KeepTrackingQueryParameters in PowerShell to opt out.

By default the crawler is page-oriented and only keeps text/html and application/xhtml+xml responses. Use RestrictToAllowedContentTypes = false in C# or -AllowAnyContentType in PowerShell when you intentionally want non-HTML responses included.

It also skips obvious asset/document URLs such as *.pdf, *.jpg, *.zip, and fonts before fetching them. Use SkipKnownAssetUrls = false in C# or -AllowAssetUrls in PowerShell when those URLs are part of the dataset you want.

If you want those assets as part of the offline dataset without treating them as pages, enable DownloadAssets in C# or -DownloadAssets in PowerShell. That saves discovered image/document URLs, stylesheet links, and CSS url(...) assets into assets/ and records them in assets.jsonl. Without DownloadAssets, assets.jsonl is still created as part of the dataset shape, but it stays empty because the crawl is page-only.

When assets are downloaded, stored HTML snapshots are also rewritten to local relative paths by default, so an <img> can point at ../assets/... instead of the original remote URL. Use RewriteAssetReferencesToLocal = false in C# or -KeepRemoteAssetUrls in PowerShell if you want to keep the original references.

Stored HTML also rewrites internal page links to local saved .html files by default, which makes the persisted crawl much more browsable offline. Use RewritePageLinksToLocal = false in C# or -KeepRemotePageUrls in PowerShell if you want to preserve original page URLs.

Pages that use <base href> are also handled correctly during crawl discovery and offline rewrite, and the saved HTML strips the original <base> tag so local links and assets do not get redirected back to the live site.

Each saved page also gets a sidecar .json manifest with its metadata, outgoing links, referenced asset URLs, and any downloaded asset file paths resolved relative to that page. The generated index.html links to those manifests directly.

Those per-page manifests now also include lightweight search metadata such as heading extraction, word counts, character counts, and a short summary snippet, and index.html surfaces the same summary information for quick offline scanning.

The persisted dataset also exports a global chunks.jsonl file with deduplicated text chunks, per-chunk summaries, heading context, normalized keywords, and relative links back to each saved page, text file, and manifest so it is easy to feed into local search or RAG tooling.

It also exports graph.json, which captures fetched pages as nodes and discovered page-to-page links as edges, including fetched/skipped/external node categories, edge relation types, in-degree/out-degree counts, and relative paths back to saved HTML and manifest files for offline analysis or navigation tooling. The generated summaries now also break those graph counts down by node category, edge relation, and skipped-node reason.

🔧 PowerShell Cmdlets

HTML/CSS/JavaScript Processing

Convert-HTMLToText - Convert markup to plain text
ConvertFrom-HtmlTable - Extract table elements into objects (supports rowspan/colspan)
ConvertFrom-HTMLAttributes - Extract elements by tag, class, id or name
ConvertFrom-HTML - Parse full documents or fragments
ConvertFrom-HtmlForm - Extract form data and structure
ConvertFrom-HtmlList - Parse list elements into structured data
ConvertFrom-HtmlMeta - Extract name/content pairs from meta tags
ConvertFrom-HtmlMicrodata - Extract structured data items (schema.org types)
ConvertFrom-HtmlOpenGraph - Extract Open Graph metadata
ConvertFrom-HtmlJsonLd - Extract JSON-LD structured data scripts
ConvertFrom-HtmlScriptData - Extract generic JSON-bearing script data such as import maps and app settings
ConvertFrom-HtmlAppState - Extract common framework state payloads such as __NEXT_DATA__
ConvertFrom-HtmlHeadLink - Extract canonical, alternate, feed, icon, manifest, and preload head links
ConvertFrom-HtmlImageCandidate - Extract image URLs and responsive srcset candidates
Select-HtmlNode - Select HtmlAgilityPack nodes with XPath or common tag/attribute predicates
Select-HtmlAttributeValue - Read HTML attribute values with fallback defaults
Select-HtmlInnerText - Read node text with optional HTML entity decoding
Select-HtmlToken - Find CSRF, XSRF, nonce, anti-forgery, and auth token values
Select-HtmlJavaScriptVariable - Find JavaScript variable declarations and assignments inside HTML script tags
ConvertFrom-JavaScriptAst - Parse JavaScript into an Acornima AST
Select-JavaScriptAstNode - Traverse Acornima AST descendants by node type
Select-JavaScriptVariable - Find JavaScript variable declarations and loose assignments by name or prefix
ConvertFrom-HtmlRscPayload - Extract Next.js inline React Server Component / React Flight payload rows
ConvertFrom-JavaScriptEndpoint - Discover likely endpoints from static JavaScript strings
ConvertFrom-HtmlLinkedJavaScriptEndpoint - Download linked scripts and discover likely endpoints from JavaScript bundles
ConvertFrom-RobotsTxt - Parse robots.txt groups, rules, crawl delays, and sitemap directives
ConvertFrom-WebManifest - Parse web app manifest JSON
ConvertFrom-WellKnownText - Parse security.txt, humans.txt, and ads.txt
Format-CSS - Pretty-print style sheets
Format-HTML - Tidy up HTML markup
Format-JavaScript - Beautify JavaScript with customizable options
Optimize-CSS - Minify style sheets
Optimize-Email - Inline CSS for email bodies
Optimize-HTML - Minify HTML
Optimize-JavaScript - Minify JavaScript

Browser Automation & Interaction

Start-HtmlBrowserSession / Invoke-HtmlRendering - Create browser sessions with authentication support
Close-HtmlBrowserSession - Dispose browser sessions
Invoke-HtmlBrowserNavigation - Navigate to different URLs
Invoke-HtmlBrowserScript - Execute JavaScript in browser context
Invoke-HtmlBrowserDomScript - Run JavaScript with AngleSharp (no browser required)
Invoke-HtmlBrowserClick - Click elements in the browser
Get-HtmlBrowserInteractable - List clickable elements
Set-HtmlBrowserInput - Set input field values
Invoke-HtmlBrowserKey - Send keyboard input such as Enter, Control+A, or ArrowDown
Invoke-HtmlBrowserHover - Hover over an element
Invoke-HtmlBrowserScroll - Scroll an element into view
Wait-HtmlBrowserReady - Wait for load state, selector, JavaScript readiness, and/or DOM stability
Find-HtmlBrowserLocator - Rank resilient selector and Playwright-style locator candidates for visible controls
Wait-HtmlBrowserContent - Wait for text or a target element state
Close-HtmlBrowserOverlay - Try common cookie/modal dismissals
Get-HtmlBrowserDiagnostics - Inspect navigator, viewport, storage, console errors, observed API calls, and WebSockets
Get-HtmlBrowserElement / Test-HtmlBrowserElement - Inspect selector matches, geometry, attributes, and state
Get-HtmlBrowserActiveElement - Inspect the focused element after clicks or keyboard navigation
Get-HtmlBrowserStorage / Set-HtmlBrowserStorage - Read or update local/session storage entries
Save-HtmlBrowserContent - Save rendered session or one-shot URL/file HTML or text to disk
Export-HtmlBrowserEvidence - Save screenshots, PDFs, rendered content, text/Markdown, optional network summary, redacted SSO handoff summary, and manifest hashes; text artifacts and manifest URLs redact common secrets by default, and visual artifacts mask common sensitive fields by default; use -Artifact for minimal packs, -NoRedaction only when exact raw proof is required, and -NoVisualMask only when image/PDF masking is not appropriate
New-HtmlBrowserProfile -Scenario / Start-HtmlBrowserSession -Scenario - Apply intent-focused browser defaults for audit proof, mailbox proof, login-protected pages, SPAs, low bandwidth, network capture, and download evidence
Start-HtmlBrowserSession -ManualLogin - Open a visible persistent session for enterprise SSO/MFA and optionally wait for a post-login selector before returning
Start-HtmlBrowserSession -PreventSsoAutoSubmit / Get-HtmlBrowserSsoHandoff -Wait - Pause and inspect SAML, WS-Federation, OAuth, or OpenID Connect handoff forms after user-attended login; assertion and token fields are redacted by default and require -IncludeSensitiveValues to reveal
Find-HtmlBrowserDataSource - Convert observed browser XHR/fetch traffic into browserless extraction sources and recipes
Find-HtmlDataSource - Discover static, app-state, JSON, and endpoint sources before opening a browser
Invoke-HtmlDataExtraction - Extract a discovered browserless source
Export-HtmlExtractionRecipe / Import-HtmlExtractionRecipe / Invoke-HtmlExtractionRecipe - Save and rerun browserless extraction recipes
Start-HtmlBrowserRecipeRecording / Stop-HtmlBrowserRecipeRecording / Export-HtmlBrowserRecipe / Optimize-HtmlBrowserRecipe / Invoke-HtmlBrowserRecipe - Record successful session actions, harden selector alternates from the live page, save JSON browser recipes, and replay them with step results, locator output, evidence packs, optional failure evidence, and recorder-side redaction for sensitive input fields
Set-HtmlBrowserSelectOption - Select dropdown options
Set-HtmlBrowserChecked - Check/uncheck checkboxes and radio buttons
Submit-HtmlBrowserForm - Submit forms

Browser Extraction Mode

Browser extraction mode is the rendering-backed path for pages where static HTML is not enough. The intended workflow is: plan the page, choose the lightest render profile that matches the problem, interact only as needed, snapshot the rendered state, inspect diagnostics, and feed the same rendered snapshot into the workbench.

Browser scenarios are a higher-level starting point for admin automation. They set practical defaults first, then profile JSON and explicit parameters can refine them. AuditProof, MailboxProof, and DownloadEvidence use a stable 1366x900 evidence viewport and DOM-ready navigation. LoginProtected extends timeouts for SSO/MFA work. SinglePageApp avoids waiting forever on busy app traffic. LowBandwidth skips heavy images, media, and fonts. NetworkCapture keeps rendered output and observed traffic together for diagnostics.

For Azure AD, Okta, ADFS, and other enterprise SSO pages, prefer a real browser session over rebuilding the login flow with Invoke-WebRequest. SAML and WS-Federation often use a hidden auto-submitting form that appears only after JavaScript, redirects, MFA, and policy checks finish, so -PreventSsoAutoSubmit can hold recognized handoff forms while Get-HtmlBrowserSsoHandoff -Wait -Timeout 60000 reports protocol, action URL, field names, original value lengths, and redacted values. OAuth and OpenID Connect redirects can also put code, state, id_token, access_token, or error fields in the current URL query or fragment; those are returned as a location handoff with the page URL and field values redacted by default. Use -IncludeSensitiveValues only for your own authorized workflow when the raw assertion or token is genuinely required.

When the site only works from a seeded real browser profile, attach to Chrome or Edge over the Chrome DevTools Protocol instead of launching a fresh Playwright browser. Start the browser yourself with a remote debugging port and the profile/history/session state you want to reuse, then pass -CdpEndpointUrl directly or save it in a profile JSON. HtmlTinkerX creates and owns only the automation page; closing the PowerShell session does not close the attached browser or profile.

$userDataPath = Join-Path $PWD 'real-chrome-user-data'
chrome.exe --remote-debugging-port=9222 --user-data-dir="$userDataPath"

New-HtmlBrowserProfile `
    -Name 'RealChromeCdp' `
    -Path '.\real-chrome-cdp.profile.json' `
    -CdpEndpointUrl 'http://127.0.0.1:9222' `
    -PreventSsoAutoSubmit | Out-Null

$session = Start-HtmlBrowserSession `
    -Url 'https://example.com/protected' `
    -ProfilePath '.\real-chrome-cdp.profile.json' `
    -NoDefault

When a captured handoff needs inspection rather than replay, start with Get-HtmlBrowserSsoHandoff -Analyze. It auto-detects SAMLResponse, id_token, access_token, authorization code, state, and RelayState fields and routes known artifacts through safe protocol summaries without returning raw assertion or token values. If you already have a saved/deserialized handoff object, pipe it to ConvertFrom-HtmlSsoHandoff for the same analysis. For deeper protocol-specific inspection, use ConvertFrom-HtmlSamlResponse for SAML assertions or ConvertFrom-HtmlJsonWebToken for OpenID Connect/OAuth tokens. These report issuer, destination or audience, validity window, status or scopes, key id, and claim or attribute names while redacting subject, assertion, and user-identifying values by default. Add -IncludeSensitiveValues only for authorized troubleshooting, and -IncludeXml or -IncludeJson only when you need decoded payloads.

Evidence screenshots and PDFs mask common sensitive fields by default, including password, token, SAML, MFA, OTP, PIN, credential, and secret inputs. Add -VisualMaskSelector for app-specific fields, change the mask color with -VisualMaskColor, or use -NoVisualMask only when the visual proof must intentionally show those fields. The older -ScreenshotMaskSelector, -ScreenshotMaskColor, and -NoScreenshotMask names remain available. Direct Save-HtmlBrowserScreenshot, Save-HtmlBrowserPdf, Save-HtmlBrowserContent, and Export-HtmlBrowserEvidence calls can reuse one-shot launch defaults such as -Scenario, -ProfilePath, -StatePath, -UserDataDirectory, -LoadState, -Proxy, and resource blocking. Screenshot and PDF calls also expose opt-in masking through -MaskSensitiveElement, -MaskSelector, and -MaskColor. BlockResourceType Document is intentionally rejected because it would abort the page itself rather than only blocking subresources.

Profile	Use when	PowerShell story	C# story
`FastStaticFallback`	static extraction is likely enough but a cheap browser fallback is useful	`Invoke-HtmlRendering -RenderProfile FastStaticFallback`	`HtmlBrowser.GetPageContentAsync(...)` with blocked heavy resources
`InteractivePage`	forms, buttons, search boxes, or reactive inputs reveal content	`Start-HtmlBrowserSession`, `Set-HtmlBrowserInput`, `Invoke-HtmlBrowserKey`, `Wait-HtmlBrowserReady`, `Wait-HtmlBrowserContent`	`HtmlBrowser.TypeInputAsync`, `PressKeysAsync`, `WaitUntilReadyAsync`, `WaitForTextAsync`
`LazyLoadedContent`	content appears after scrolling or delayed viewport work	`Invoke-HtmlRendering -RenderProfile LazyLoadedContent -AutoScroll`	`HtmlBrowser.CreateSnapshotAsync(...)` after scroll/wait helpers
`AppShell`	the original HTML is a thin JavaScript shell	`Invoke-HtmlRendering -RenderProfile AppShell -Snapshot`	planner recommends `HtmlRenderProfile.AppShell` before workbench analysis
`LoginProtected`	storage state or a visible login session is needed	`Start-HtmlBrowserSession`, `Export-HtmlBrowserState`, `-StatePath` / `-UserDataDirectory`	`HtmlBrowser.ExportBrowserStateAsync`, `ImportBrowserStateAsync`, and persistent launch options
`NetworkCapture`	API calls, response bodies, console errors, or WebSockets explain the page	`-IncludeNetworkLog -IncludeResponseBody -RedactResponseBody`	`HtmlBrowser.GetNetworkLog`, `GetDiagnosticsAsync`
`LowBandwidth`	bandwidth is constrained or media/fonts/styles are not needed	`Invoke-HtmlRendering -RenderProfile LowBandwidth`	browser content with aggressive resource blocking
`HeavyDynamicPage`	legacy broad dynamic-page profile is still needed	`Invoke-HtmlRendering -RenderProfile HeavyDynamicPage`	existing high-wait dynamic rendering behavior

# Hydrated app page with visible-only interactions before extraction
$snapshot = Invoke-HtmlRendering -Url 'https://example.com/app' `
    -RenderProfile InteractivePage `
    -DismissText 'Accept' `
    -ClickText 'Load more' `
    -WaitForSelector 'main' `
    -Selector 'main' `
    -Snapshot

# Lazy-loaded content that appears while scrolling
$content = Invoke-HtmlRendering -Url 'https://example.com/catalog' `
    -RenderProfile LazyLoadedContent `
    -WaitForSelector '.product-card' `
    -Selector '.product-grid'

# JavaScript app shell with parsed snapshot data
$app = Invoke-HtmlRendering -Url 'https://example.com/dashboard' `
    -RenderProfile AppShell `
    -WaitForSelector 'main' `
    -Snapshot `
    -IncludeLinkedScripts `
    -IncludeStaticRenderedComparison

# Network-focused pass for observed API calls and console errors
$network = Invoke-HtmlRendering -Url 'https://example.com/search' `
    -RenderProfile NetworkCapture `
    -WaitForSelector '#results' `
    -Snapshot `
    -IncludeNetworkLog `
    -IncludeResponseBody `
    -RedactResponseBody

# Session-style interaction with readiness waits and paced typing
$session = Start-HtmlBrowserSession -Url 'https://example.com/search' -Scenario SinglePageApp
Close-HtmlBrowserOverlay -Session $session
Set-HtmlBrowserInput -Session $session -Selector 'input[type=search]' -Value 'HtmlTinkerX' -Type -DelayMs 25
Invoke-HtmlBrowserKey -Session $session -Selector 'input[type=search]' -Key 'Enter'
Wait-HtmlBrowserReady -Session $session -Selector 'main' -Stable
Wait-HtmlBrowserContent -Session $session -Text 'Results' -Selector 'main'
Wait-HtmlBrowserContent -Session $session -Element -Selector 'main' -Visible -InViewport
$items = Get-HtmlBrowserElement -Session $session -Selector '.result' -VisibleOnly -IncludeAttributes
$items | Select-Object Text, Selector, Visible, InViewport, Width, Height
Get-HtmlBrowserContent -Session $session -Selector 'main' -AsText
$allTitles = Get-HtmlBrowserContent -Session $session -Selector '.result-title' -All -AsText
$active = Get-HtmlBrowserActiveElement -Session $session -IncludeAttributes
Set-HtmlBrowserStorage -Session $session -Scope Local -Key extractionMode -Value browser
$storage = Get-HtmlBrowserStorage -Session $session -Scope All
Save-HtmlBrowserContent -Session $session -Selector 'main' -OutFile .\rendered-main.html
$evidence = Export-HtmlBrowserEvidence -Session $session -OutFolder .\evidence\search -NetworkSummary -SsoHandoffSummary -VisualMaskSelector '.account-secret'
$diagnostics = Get-HtmlBrowserDiagnostics -Session $session
$diagnostics.ConsistencyWarnings
$diagnostics.ObservedApiCalls
$apiSource = Find-HtmlBrowserDataSource -Session $session -IncludeResponseBody | Select-Object -First 1
$apiSource | Export-HtmlExtractionRecipe -Path .\observed-api.recipe.json -IncludeRawContent
$apiResult = $apiSource | Invoke-HtmlDataExtraction
Close-HtmlBrowserSession -Session $session

# Record a manual session into a replayable browser recipe
$recorded = Start-HtmlBrowserSession -Url 'https://example.com/search' -Scenario SinglePageApp
Start-HtmlBrowserRecipeRecording -Session $recorded -Name 'SearchProof' -IncludeCurrentUrl
Set-HtmlBrowserInput -Session $recorded -Selector 'input[type=search]' -Value 'HtmlTinkerX'
Invoke-HtmlBrowserClick -Session $recorded -Text 'Search' -Exact
Wait-HtmlBrowserContent -Session $recorded -Text 'Results' -Selector 'main'
Export-HtmlBrowserEvidence -Session $recorded -OutFolder .\evidence\recorded-search -BaseFileName search-proof -Artifact Html,Text -NoManifest
Stop-HtmlBrowserRecipeRecording -Session $recorded -Path .\search-proof.browser.recipe.json -VariableTemplatePath .\search-proof.browser.variables.json -HardenSelectors -HardeningReportPath .\search-proof.browser.hardening.json
Close-HtmlBrowserSession -Session $recorded
Invoke-HtmlBrowserRecipe -Path .\search-proof.browser.recipe.json

Browser recipe recording redacts values entered into selectors that look sensitive, such as password, token, SAML, MFA, OTP, or secret fields. The recipe step keeps `ValueRedacted`, `ValueRedactionReason`, and `ValueVariable` so you can see that a value was intentionally omitted and provide it at replay time. Keep real credentials in a vault or provide them at runtime instead of committing them into recipe JSON.

```powershell
$validation = Test-HtmlBrowserRecipe -Path .\search-proof.browser.recipe.json
$validation.RequiredVariables
$validation.VariableTemplate
$validation.BlockingIssues | Format-Table Severity, StepIndex, Action, Property, Message, SuggestedFix, SuggestedCommand -AutoSize

Test-HtmlBrowserRecipe -Path .\search-proof.browser.recipe.json -StrictPreflight -ThrowOnFailure

$reviewSession = Start-HtmlBrowserSession -Url 'https://example.com/search' -Scenario SinglePageApp
$review = Optimize-HtmlBrowserRecipe -Session $reviewSession -Path .\search-proof.browser.recipe.json -OutPath .\search-proof.browser.hardened.recipe.json -ReportPath .\search-proof.browser.hardening.json
$review.Steps | Where-Object Changed | Select-Object StepIndex, Action, AddedAlternates, Reason
Close-HtmlBrowserSession -Session $reviewSession

$secret = Read-Host -AsSecureString -Prompt 'Portal password'
Invoke-HtmlBrowserRecipe -Path .\search-proof.browser.recipe.json -VariablePath .\search-proof.browser.variables.json -Variable @{ password = $secret }

Test-HtmlBrowserRecipe returns CI-friendly fields such as Passed, BlockingIssues, BlockingIssueCount, RecommendedExitCode, and Summary. Each issue includes SuggestedFix, SuggestedCommand, and DocumentationHint, so the next repair step is visible without opening DevTools or guessing which browser cmdlet applies. Use -StrictPreflight when CI or a scheduled run should also block on warnings such as ContinueOnError, long fixed waits, or sensitive selectors, and add -ThrowOnFailure when the validation command itself should fail the job. Invoke-HtmlBrowserRecipe also preflights before opening a browser. Missing runtime variables, empty steps, invalid timeouts, and missing selectors return a failed recipe run result with SkippedBeforeExecution, Validation, FailureSummary, and SuggestedCommand populated, so scheduled jobs can fail fast without launching Chromium. Use -SkipPreflight only when you intentionally want to reproduce the old replay-time failure path.

$browserProfilePath = Join-Path$ PWD 'browser-profile.json' $userDataPath = Join-Path$ PWD 'browser-user-data' New-HtmlBrowserProfile -Name 'EnterpriseLogin' -Scenario LoginProtected -Path $browserProfilePath -UserDataDirectory$ userDataPath $loginSession = Start-HtmlBrowserSession -Url 'https://example.com/protected' -ProfilePath$ browserProfilePath -ManualLogin -LoginSuccessSelector 'main' Export-HtmlBrowserState -Session $loginSession -Path (Join-Path$ PWD 'browser-state.json') Export-HtmlBrowserEvidence -Session $loginSession -OutFolder (Join-Path$ PWD 'evidence\login') -NetworkSummary -SsoHandoffSummary Close-HtmlBrowserSession -Session $loginSession

Pause an enterprise SSO handoff form long enough to inspect it safely

$ssoSession = Start-HtmlBrowserSession -Url 'https://example.com/protected' -Scenario LoginProtected -Visible -ManualLogin -PreventSsoAutoSubmit try { Get-HtmlBrowserSsoHandoff -Session$ ssoSession -Wait -Timeout 60000 Get-HtmlBrowserSsoHandoff -Session $ssoSession -Analyze Export-HtmlBrowserEvidence -Session$ ssoSession -OutFolder (Join-Path $PWD 'evidence\sso') -SsoHandoffSummary Get-HtmlBrowserSsoHandoff -Session$ ssoSession -IncludeSensitiveValues } finally { Close-HtmlBrowserSession -Session $ssoSession }

Ask the planner which browser/content profile to use

$plan = Test-HtmlExtractionPlan -Url 'https://example.com/app'$ plan.SuggestedProfileCommand $plan | Get-HtmlExtractionProfile


Run [Examples/Example-BrowserExtractionModeLocal.ps1](Examples/Example-BrowserExtractionModeLocal.ps1) for an offline, self-contained version of the story. It stubs a local page and API with `Register-HtmlRoute`, then runs:

```powershell
Start-HtmlBrowserSession
Invoke-HtmlNavigation
Close-HtmlBrowserOverlay
Set-HtmlBrowserInput -Type
Invoke-HtmlBrowserKey
Invoke-HtmlBrowserClick -Text
Wait-HtmlBrowserReady
Wait-HtmlBrowserContent
Get-HtmlBrowserElement
Test-HtmlBrowserElement
Get-HtmlBrowserActiveElement
Get-HtmlBrowserStorage
Set-HtmlBrowserStorage
Save-HtmlBrowserContent
Export-HtmlBrowserEvidence
Get-HtmlBrowserDiagnostics
Compare-HtmlStaticRendered
Invoke-HtmlPageWorkbench -RenderedSnapshot
Export-HtmlBrowserState

Run Examples/Example-BrowserEvidenceProofPack.ps1 for a self-contained proof-pack workflow with a reusable browser profile, persistent user-data directory, readiness wait, masked screenshot evidence, and evidence manifest.

Run Examples/Example-BrowserManualLogin.ps1 for a visible enterprise login workflow that reuses a persistent profile, waits for a post-login selector, exports browser state, and writes an evidence pack.

Run Examples/Example-BrowserSsoHandoff.ps1 for an offline SAML handoff workflow. It waits for a delayed auto-submitting handoff form, reports redacted field values by default, and demonstrates the explicit reveal switch without using a real tenant.

Run Examples/Example-BrowserNetworkToRecipe.ps1 for an offline network-to-recipe workflow. It observes a browser fetch, captures the JSON body, turns it into a browserless data source, exports a recipe, and replays the recipe without navigating the page again.

Run Examples/Example-BrowserRecipeRecording.ps1 for an offline recorder workflow. It performs normal HtmlTinkerX session actions, redacts a password-like recorded input, records a minimal HTML/text evidence pack, saves the successful steps as a browser recipe, then replays that recipe in a fresh browser session with the same evidence artifact choices.

Run Examples/Example-BrowserFailureEvidenceAndLocators.ps1 for an offline locator-discovery and failure-evidence workflow. It ranks locator candidates, uses the best candidate for a click, then intentionally fails a wait with -OnFailureEvidence so the screenshot, HTML, text, network summary, failure context, and manifest can be inspected.

Run Examples/Example-BrowserRecipe.ps1 for an offline browser recipe workflow. It writes a JSON recipe, replays it with Invoke-HtmlBrowserRecipe, records locator candidates in the step results, and exports an evidence manifest.

The matching C# example is Sources/HtmlTinkerX.Examples/BrowserExtractionModeExample.cs. It follows the same sequence with HtmlBrowser.OpenSessionAsync, RegisterRouteAsync, TypeInputAsync, PressKeysAsync, WaitForTextAsync, WaitForElementStateAsync, GetElementsAsync, TestElementAsync, GetActiveElementAsync, GetStorageAsync, SetStorageAsync, SaveContentAsync, GetDiagnosticsAsync, CreateSnapshotAsync, HtmlPageWorkbench.AnalyzeAsync, and ExportBrowserStateAsync.

Diagnostics are intentionally diagnostics, not stealth. Get-HtmlBrowserDiagnostics reports browser/runtime consistency, storage keys, failed or blocked requests, observed API calls, WebSockets, and console errors so you can understand why extraction did or did not work. It does not try to hide automation, defeat access controls, or bypass site protections.

Browser extraction should stay browser-last where possible. Prefer Test-HtmlExtractionPlan, static parsers, Find-HtmlDataSource, Invoke-HtmlDataExtraction, endpoint discovery, linked JavaScript inspection, crawl profiles, and static-vs-rendered comparison before adding click-through automation.

See Examples/Example-BrowserExtractionMode.ps1 for longer workflow samples covering product search, lazy-loaded pages, app-shell snapshots, login state reuse, network capture, diagnostics, and docs crawling.

Browserless Extraction Mode

Browserless extraction mode is the Playwright-free path for pages where useful data already exists in static HTML, embedded framework state, JSON-LD, script data, or low-risk API endpoints. It does not imitate a browser; it inspects legitimate page artifacts and only performs direct HTTP reads when the source is a low-risk GET candidate or the caller explicitly opts in.

$sources = Find-HtmlDataSource -Url 'https://example.com/products' -IncludeLinkedScripts -DirectOnly

$result = $sources |
    Where-Object Kind -In 'AppState', 'JsonLd', 'ScriptData' |
    Select-Object -First 1 |
    Invoke-HtmlDataExtraction

$result.Mode
$result.Source.Kind
$result.Items | Select-Object Name, Type, Path, Value

$source = $sources | Select-Object -First 1
$source | Export-HtmlExtractionRecipe -Path .\product.recipe.json -IncludeRawContent
Import-HtmlExtractionRecipe -Path .\product.recipe.json | Invoke-HtmlExtractionRecipe

Endpoint sources are guarded by default:

$api = Find-HtmlDataSource -Url 'https://example.com/products' -IncludeLinkedScripts |
    Where-Object { $_.Kind -eq 'ApiEndpoint' -and $_.CanExtractDirectly } |
    Select-Object -First 1

$api | Invoke-HtmlDataExtraction -AllowHttpFetch

The matching C# story uses the same core:

var sources = await HtmlBrowserlessExtraction.DiscoverAsync(
    html,
    new HtmlBrowserlessDiscoveryOptions {
        BaseUri = new Uri("https://example.com/products"),
        IncludeLinkedScripts = true,
        DirectOnly = true
    });

var source = sources.First();
var result = await HtmlBrowserlessExtraction.ExtractAsync(source);
var recipe = HtmlBrowserlessExtraction.CreateRecipe(source, includeRawContent: true);
var replayed = await HtmlBrowserlessExtraction.ExtractRecipeAsync(recipe);

Run Examples/Example-BrowserlessExtraction.ps1 or the C# BrowserlessExtractionExample for a self-contained app-state extraction and recipe round trip.

Screenshots & Media

Save-HtmlBrowserScreenshot - Capture page screenshots with advanced options, including explicit sensitive-field masking
Save-HtmlBrowserPdf - Generate PDFs from rendered pages
Start-HtmlBrowserVideoCapture / Stop-HtmlBrowserVideoCapture - Record browser sessions

Network & Debugging

Get-HtmlBrowserNetworkLog - View captured network requests and responses
Get-HtmlBrowserConsoleLog - Retrieve browser console messages
Export-HtmlBrowserHar - Export network traffic to HAR files
Start-HtmlBrowserTracing / Stop-HtmlBrowserTracing - Record Playwright traces
Register-HtmlRoute / Unregister-HtmlRoute - Intercept and mock requests
Test-HtmlBrowser - Comprehensive browser testing for errors, performance, and resources
Clear-HtmlBrowserCache - Clean Playwright browser downloads

Cookies & State Management

Get-HtmlBrowserCookie - Retrieve cookies from sessions
Set-HtmlBrowserCookie - Add cookies to sessions
New-HtmlBrowserCookie - Create cookie objects
Export-HtmlBrowserState / Import-HtmlBrowserState - Save/restore browser state
Export-HtmlBrowserSession / Import-HtmlBrowserSession - Session state management

Content & Resources

Get-HTMLResource - Extract script and CSS resources
Invoke-HTMLCrawl - Crawl sites offline with optional browser rendering : supports sitemap discovery, robots.txt, filtering, auth, and selector-based extraction
Save-HtmlBrowserAttachment - Download files from pages
Get-HtmlBrowserContent - Retrieve page content
Get-HtmlBrowserFormField - Extract form field information
Get-HtmlBrowserLoginForm - Detect login forms
Export-HTMLOutline - Generate document outlines
Show-HtmlBrowserHar - Visualize HAR files
Compare-HTML - Compare HTML documents
Measure-HTMLDocument - Analyze document metrics

JavaScript AST parsing

HtmlTinkerX currently uses Jint 4.x, which uses Acornima for JavaScript parsing. Older Jint 3.x builds used Esprima, so older examples that referenced [Esprima.JavaScriptParser] should be translated to the Acornima surface:

$ast = ConvertFrom-JavaScriptAst -Content 'const settings = { apiKey: "abc" };'
$ast | Select-JavaScriptAstNode -Type ObjectExpression
$ast | Select-JavaScriptVariable -Name api -Contains

$config = Select-JavaScriptVariable -Source '$Config = { sCtx: "abc", enabled: !0 }' -Name '$Config'
$config.Value['sCtx']

Select-JavaScriptVariable -Source 'window.$Config = { auth: { sCtx: "abc" } }' -Name '$Config' -PropertyPath auth.sCtx

Select-HtmlJavaScriptVariable -Content $html -Name '$Config' -PropertyPath auth.sCtx

CSS and HTML workflow audits

Use the CSS query cmdlets to inspect generated stylesheets for theme tokens, declarations, asset URLs, media-query overrides, and selector specificity:

$css = @'
:root { --brand-color: #0369a1; }
.btn { color: var(--brand-color); background-image: url("/img/button.png"); }
@media (min-width: 40rem) {
    .btn.primary { color: white !important; }
}
'@

Select-CssRule -Content $css -Selector '.btn'
Select-CssDeclaration -Content $css -Property color
Get-CssVariable -Content $css -Name '--brand-color'
ConvertFrom-CssUrl -Content $css -BaseUrl 'https://example.org/app/'
Measure-CssSpecificity -Selector '#app .btn:hover'

Use the HTML workflow bridges to inspect scripts, page assets, and common compatibility issues without launching a browser:

$html = @'
<link rel="stylesheet" href="/css/site.css">
<link rel="manifest" href="/site.webmanifest">
<script type="application/ld+json">{"name":"schema"}</script>
<script type="module" src="/app/module.js"></script>
<img id="logo" src="/img/logo.png">
<label for="missing"></label>
<input id="email" name="email">
'@

Select-HtmlScript -Content $html -BaseUrl 'https://example.org/' -JavaScript
Select-HtmlAsset -Content $html -BaseUrl 'https://example.org/'
Measure-HtmlCompatibility -Content $html -BaseUrl 'https://example.org/'

React Server Component payload extraction

Next.js can inline React Flight payload instructions in scripts that push data into self.__next_f. Use ConvertFrom-HtmlRscPayload to inspect that server-rendered app state through stable HtmlTinkerX objects without exposing framework dependency types:

$rows = ConvertFrom-HtmlRscPayload -Content $html
$rows | Where-Object IsJson | Select-Object Id, Tag, Kind, Data

$payloads = ConvertFrom-HtmlRscPayload -Content $html -RawPayload
$document = ConvertFrom-HtmlRscPayload -Content $html -AsDocument

This is a static extractor. It does not hydrate React, execute application JavaScript, or resolve client module references.

Modern page parsing helpers

Several cmdlets expose stable HtmlTinkerX models for common data embedded in modern pages:

$jsonLd = ConvertFrom-HtmlJsonLd -Content $html
$scriptData = ConvertFrom-HtmlScriptData -Content $html
$appState = ConvertFrom-HtmlAppState -Content $html
$headLinks = ConvertFrom-HtmlHeadLink -Content $html -BaseUrl https://example.org/
$images = ConvertFrom-HtmlImageCandidate -Content $html -BaseUrl https://example.org/
$tokens = Select-HtmlToken -Content $html
$rows = ConvertFrom-HtmlRscPayload -Content $html
$endpoints = ConvertFrom-JavaScriptEndpoint -Content $html -Html
$linkedEndpoints = ConvertFrom-HtmlLinkedJavaScriptEndpoint -Url https://example.org/
$robotsRules = ConvertFrom-RobotsTxt -Content $robots -BaseUrl https://example.org/robots.txt
$manifest = ConvertFrom-WebManifest -Content $manifestJson -BaseUrl https://example.org/manifest.webmanifest
$security = ConvertFrom-WellKnownText -Content $securityTxt -Kind SecurityTxt -BaseUrl https://example.org/.well-known/security.txt

$tokens | Where-Object Source -in 'Input', 'Meta'
$endpoints | Where-Object Method -eq POST

The same parsing layer is available from C# through HtmlTinkerX:

var jsonLd = HtmlJsonLdParser.Parse(html);
var scriptData = HtmlScriptDataParser.Parse(html);
var appState = HtmlAppStateParser.Parse(html);
var headLinks = HtmlHeadLinkParser.Parse(html, new Uri("https://example.org/"));
var images = HtmlImageCandidateParser.Parse(html, new Uri("https://example.org/"));
var tokens = HtmlTokenParser.Parse(html);
var reactFlight = HtmlReactFlightParser.Parse(html);
var endpoints = HtmlJavaScriptEndpointParser.ParseHtml(html);
var linkedEndpoints = await HtmlLinkedJavaScriptEndpointParser.ParseUrlAsync("https://example.org/");
var robotsRules = HtmlRobotsParser.Parse(robots, new Uri("https://example.org/robots.txt"));
var manifest = HtmlWebManifestParser.Parse(manifestJson, new Uri("https://example.org/manifest.webmanifest"));
var security = HtmlWellKnownParser.Parse(securityTxt, "security.txt", new Uri("https://example.org/.well-known/security.txt"));

Runnable examples are available in Examples\Example-ModernParsing.ps1 and Sources\HtmlTinkerX.Examples\ModernParsingExample.cs.

These helpers are static parsers. They do not execute application JavaScript and they return PSParseHTML/HtmlTinkerX objects rather than exposed dependency implementation types.

For lower-level exploration, the packaged PowerShell module exposes a small dependency type accelerator surface through its AssemblyLoadContext boundary: common HtmlAgilityPack document/node types, core Acornima AST document types, and public dependency enums.

$script = ConvertFrom-JavaScriptAst -Content 'const answer = 42;'
$script | Select-JavaScriptAstNode -Type VariableDeclaration
$script | Select-JavaScriptAstNode -Type Script -IncludeRoot
[HtmlAgilityPack.HtmlNodeType]::Element

🎯 C# API Reference

Core Classes

HtmlParser

// Parse with different engines
var doc = HtmlParser.ParseWithAngleSharp(html);
var doc2 = HtmlParser.ParseWithHtmlAgilityPack(html);

// Extract tables with detailed information
var tables = HtmlParser.ParseTablesWithAngleSharpDetailed(html);
var tables2 = HtmlParser.ParseTablesWithHtmlAgilityPack(html);

// Parse from URLs
var urlDoc = await HtmlParser.ParseUrlWithAngleSharpAsync("https://example.com");

HtmlFormatter

// Format different resource types
string formattedHtml = HtmlFormatter.FormatHtml(html);
string formattedCss = HtmlFormatter.FormatCss(css);
string formattedJs = HtmlFormatter.FormatJavaScript(javascript);

// Custom JavaScript formatting options
var options = new BeautifierOptions {
    IndentSize = 2,
    BraceStyle = BraceStyle.Expand
};
string customJs = HtmlFormatter.FormatJavaScript(javascript, options);

// Async operations
string formatted = await HtmlFormatter.FormatHtmlAsync(html);

HtmlOptimizer

// Minify resources
string minifiedHtml = HtmlOptimizer.OptimizeHtml(html, cssDecodeEscapes: false);
string minifiedCss = HtmlOptimizer.OptimizeCss(css);
string minifiedJs = HtmlOptimizer.OptimizeJavaScript(javascript);

// File operations
string optimizedFile = await HtmlOptimizer.OptimizeHtmlFileAsync(
    "input.html",
    cssDecodeEscapes: false);
await File.WriteAllTextAsync("output.html", optimizedFile);

HtmlBrowser (Browser Automation)

// Create browser sessions
await using var session = await HtmlBrowser.OpenSessionAsync("https://example.com");

// Form-based authentication
var formLogin = new HtmlFormLogin
{
    LoginUrl = "https://example.com/login",
    UsernameSelector = "#username",
    PasswordSelector = "#password",
    SubmitSelector = "button[type='submit']"
};
await using var authSession = await HtmlBrowser.OpenSessionAsync(
    "https://example.com/protected",
    username: "user",
    password: "pass",
    formLogin: formLogin
);

// Screenshots
await HtmlBrowser.CaptureScreenshotAsync(session.Page, "screenshot.png");
await HtmlBrowser.CaptureScreenshotAsync(
    session.Page,
    "full.png",
    new ScreenshotOptions { FullPage = true });

// PDF generation from an already-loaded page
await HtmlBrowser.SavePagePdfAsync(
    session.Page,
    "document.pdf",
    new HtmlBrowserPdfOptions(
        format: PdfPageFormat.A4,
        printBackground: true));

// High-throughput PDF generation with warm Chromium reuse and isolated contexts
await using var pdfRenderer = new HtmlBrowserPdfRenderer(
    new HtmlBrowserPdfRendererOptions(
        minimumBrowserInstances: 1,
        maximumBrowserInstances: 4,
        maximumQueuedCaptures: 32));
await pdfRenderer.PreWarmAsync();

var pdfRequest = new HtmlBrowserPdfRequest(
    HtmlBrowserPdfSource.FromHtml(
        "<main><h1>Quarterly report</h1></main>",
        new Uri("https://reports.example.com/assets/")),
    pdfOptions: new HtmlBrowserPdfOptions(
        format: PdfPageFormat.A4,
        printBackground: true,
        tagged: true,
        outline: true));

HtmlBrowserPdfResult pdfResult = await pdfRenderer.CaptureAsync(pdfRequest);
await File.WriteAllBytesAsync("quarterly.pdf", pdfResult.PdfBytes);

// Navigation
await HtmlBrowser.NavigateAsync(session, "https://example.com/page2");

// JavaScript execution
string? title = await HtmlBrowser.EvaluateAsync<string>(session, "document.title");

// Element interaction
await HtmlBrowser.ClickSelectorAsync(session, "#button");
await HtmlBrowser.FillInputAsync(session, "#username", "user");
await HtmlBrowser.ClickSelectorAsync(session, "#loginForm button[type='submit']");

HtmlBrowserPdfRenderer accepts URL, HTML-string, and file sources. Each capture gets a fresh context while the bounded pool reuses Chromium processes, applies finite prewarm and pre-navigation setup deadlines, origin-scoped headers/storage, cookie-scoped authentication, viewport and device emulation, readiness and print options, streams PDF output through a configurable 128 MiB default limit, and returns lifecycle diagnostics with the PDF bytes. HTML-string requests need an HTTP/HTTPS base URI when using headers or web storage. Capture fails if Chromium rejects a requested storage entry instead of continuing with missing state. PDF capture is Chromium-only; Firefox and WebKit requests fail before launch.

The renderer validates HTTPS and blocks private-network HTTP(S)/WS(S) and arbitrary file requests by default. Its browser-slot proxy connects to the exact DNS address accepted by policy, while canonical path checks reject file symlink escapes. Configure HtmlBrowserNetworkPolicy with explicit internal hosts or file roots when the caller and resources are trusted. A caller-supplied proxy requires explicit private-network mode because that proxy owns DNS and outbound enforcement. These controls do not replace container or host egress policy for services that accept untrusted input.

HtmlUtilities

// Convert HTML to plain text
string plainText = HtmlParserToText.ConvertToText(html);

// HTTP client operations
HttpClient httpClient = HtmlHttpClientFactory.Shared;
string content = await httpClient.GetStringAsync("https://example.com");

PreMailerClient

// Email optimization
PreMailerResult inlineResult = PreMailerClient.MoveCssInline(emailHtml, new PreMailerOptions());
string inlinedHtml = inlineResult.Html;

var preMailerOptions = new PreMailerOptions { DownloadRemoteCss = true };
PreMailerResult remoteCssResult = await PreMailerClient.MoveCssInlineAsync(emailHtml, preMailerOptions);
string optimized = remoteCssResult.Html;

Extension Methods

HtmlParserExtensions

// Quick element queries
var elements = HtmlParserExtensions.GetElements(html, className: "class-name");
var byId = HtmlParserExtensions.GetElements(html, id: "element-id");
var byTag = HtmlParserExtensions.GetElements(html, tag: "p");

📚 Examples

PowerShell Examples

Table Extraction

# Extract tables from Wikipedia
$tables = ConvertFrom-HtmlTable -Url 'https://en.wikipedia.org/wiki/PowerShell'
$tables[0] | Format-Table -AutoSize

# Parse local HTML file
$tables = ConvertFrom-HtmlTable -Path './data.html'
foreach ($table in $tables) {
    $table | Export-Csv "table_$($tables.IndexOf($table)).csv" -NoTypeInformation
}

Resource Optimization

# Format and minify HTML
$formatted = Format-HTML -Path './messy.html'
$minified = Optimize-HTML -Content $formatted -OutputFile './clean.min.html'

# Optimize JavaScript with custom options
$js = Format-JavaScript -Path './script.js' -IndentSize 2 -BraceStyle Expand
Optimize-JavaScript -Content $js -OutputFile './script.min.js'

# Email optimization
$emailHtml = Get-Content './newsletter.html' -Raw
$optimized = Optimize-Email -Body $emailHtml -UseEmailFormatter -DownloadRemoteCss

Browser Automation

# Authenticated session with form login
$cred = Get-Credential
$session = Start-HtmlBrowserSession -Url 'https://example.com/protected' `
    -Credential $cred `
    -LoginUrl 'https://example.com/login' `
    -UsernameSelector 'input[name=username]' `
    -PasswordSelector 'input[name=password]' `
    -SubmitSelector 'button[type=submit]'

# Take screenshots with different options
Save-HtmlBrowserScreenshot -Session $session -OutFile 'full-page.png' -Full
Save-HtmlBrowserScreenshot -Session $session -OutFile 'element.png' -ElementSelector '#content'
Save-HtmlBrowserScreenshot -Session $session -OutFile 'highlighted.png' -HighlightSelector '.important'

# Download files
Save-HtmlBrowserAttachment -Session $session -Path './downloads' -Filter '.pdf'

# Network monitoring
Start-HtmlBrowserTracing -Session $session
Invoke-HtmlBrowserNavigation -Session $session -Url 'https://example.com/api/data'
Stop-HtmlBrowserTracing -Session $session -OutFile 'trace.zip'
Export-HtmlBrowserHar -Session $session -OutFile 'network.har'

Close-HtmlBrowserSession -Session $session

C# Examples

Document Processing

using AngleSharp.Dom;
using HtmlTinkerX;

// Parse and process HTML
string html = await File.ReadAllTextAsync("document.html");
var document = HtmlParser.ParseWithAngleSharp(html);

// Extract specific elements
var links = document.QuerySelectorAll("a[href]");
var images = document.QuerySelectorAll("img[src]");

// Extract tables with detailed information
var tables = HtmlParser.ParseTablesWithAngleSharpDetailed(html);
foreach (var table in tables)
{
    Console.WriteLine($"Table has {table.Metadata.RowCount} rows and {table.Metadata.ColumnCount} columns");
    foreach (var row in table.Data)
    {
        Console.WriteLine(string.Join(" | ", row.Values));
    }
}

Resource Optimization

// Format resources
string formattedHtml = HtmlFormatter.FormatHtml(html);
string formattedCss = HtmlFormatter.FormatCss(css);

// Custom JavaScript formatting
var jsOptions = new BeautifierOptions
{
    IndentSize = 4,
    BraceStyle = BraceStyle.Collapse,
    PreserveNewlines = true
};
string formattedJs = HtmlFormatter.FormatJavaScript(javascript, jsOptions);

// Minification
string minifiedHtml = HtmlOptimizer.OptimizeHtml(html, cssDecodeEscapes: false);
string minifiedCss = HtmlOptimizer.OptimizeCss(css);
string minifiedJs = HtmlOptimizer.OptimizeJavaScript(javascript);

// Email optimization
string emailBody = await File.ReadAllTextAsync("newsletter.html");
var preMailerOptions = new PreMailerOptions { DownloadRemoteCss = true };
PreMailerResult preMailerResult = await PreMailerClient.MoveCssInlineAsync(emailBody, preMailerOptions);
string inlined = preMailerResult.Html;

Browser Automation

// Basic browser session
await using var session = await HtmlBrowser.OpenSessionAsync("https://example.com");

// Authenticated session
var formLogin = new HtmlFormLogin
{
    LoginUrl = "https://example.com/login",
    UsernameSelector = "#username",
    PasswordSelector = "#password",
    SubmitSelector = "#login-button"
};
await using var authSession = await HtmlBrowser.OpenSessionAsync(
    "https://example.com/protected",
    username: "username",
    password: "password",
    formLogin: formLogin
);

// Interact with the page
await HtmlBrowser.FillInputAsync(session, "#search", "query");
await HtmlBrowser.ClickSelectorAsync(session, "#search-button");
await Task.Delay(2000); // Wait for results

// Capture results
await HtmlBrowser.CaptureScreenshotAsync(session.Page, "results.png");
var consoleMessages = HtmlBrowser.GetConsoleLog(session);
foreach (var message in consoleMessages)
{
    Console.WriteLine($"{message.Type}: {message.Text}");
}

// Download files
await foreach (string download in HtmlBrowser.SavePageDownloadsAsync(session.Page, "./downloads", ".pdf"))
{
    Console.WriteLine($"Downloaded {download}");
}

🧪 Browser Testing & Network Monitoring

PSParseHTML now includes comprehensive browser testing capabilities for checking network requests, CSS resources, console errors, and performance metrics. This feature uses strongly-typed classes instead of dictionaries for better IntelliSense and type safety.

PowerShell Browser Testing

Basic Testing

# Run a comprehensive test on a URL
$result = Test-HtmlBrowser -Url 'https://example.com'

# Test a local HTML file
$result = Test-HtmlBrowser -Path 'C:\MyProject\index.html'

# Check if test passed (no errors or failed requests)
if ($result.Passed) {
    Write-Host "✅ All tests passed!"
} else {
    Write-Host "❌ Issues found: $($result.Summary)"
}

# View detailed results
Write-Host "Total Requests: $($result.TotalRequests)"
Write-Host "Failed Requests: $($result.FailedRequestCount)"
Write-Host "Console Errors: $($result.ErrorCount)"
Write-Host "Console Warnings: $($result.WarningCount)"

Testing Local HTML Files

# Test local HTML files created by HTMLForgeX or other tools
$htmlFile = "C:\Projects\MyReport\report.html"
$result = Test-HtmlBrowser -Path $htmlFile

# Check for JavaScript errors in local file
$errors = Test-HtmlBrowser -Path $htmlFile -ErrorsOnly
if ($errors.Count -gt 0) {
    Write-Host "Found $($errors.Count) JavaScript errors:"
    $errors | ForEach-Object {
        Write-Host "  - $($_.Text) at $($_.FullLocation)"
    }
}

# Test CSS loading in local file
$cssCheck = Test-HtmlBrowser -Path $htmlFile -CssResource 'styles.css'
if ($cssCheck) {
    Write-Host "CSS loaded successfully in $($cssCheck.Duration.TotalMilliseconds)ms"
}

# Test with visible browser (not headless) for debugging
$result = Test-HtmlBrowser -Path $htmlFile -Headless:$false

Testing for Console Errors

# Get only console errors
$errors = Test-HtmlBrowser -Url 'https://example.com' -ErrorsOnly

foreach ($error in $errors) {
    Write-Host "Error: $($error.Text)"
    Write-Host "  Location: $($error.FullLocation)"
    Write-Host "  Severity: $($error.SeverityLevel)"

    if ($error.StackTrace) {
        Write-Host "  Stack: $($error.StackTrace)"
    }
}

CSS Resource Testing

# Check if a specific CSS file is loaded
$cssResource = Test-HtmlBrowser -Url 'https://example.com' -CssResource 'styles.css'

if ($cssResource) {
    Write-Host "CSS found: $($cssResource.Url)"
    Write-Host "Load time: $($cssResource.Duration.TotalMilliseconds)ms"
    Write-Host "Size: $($cssResource.TransferSize) bytes"
    Write-Host "From cache: $($cssResource.ServedFromCache)"
}

Performance Testing

# Get performance metrics only
$metrics = Test-HtmlBrowser -Url 'https://example.com' -PerformanceOnly

# Display performance report
Write-Host $metrics.GetReport()

# Access specific metrics
Write-Host "Page Load Time: $($metrics.TotalLoadTime.TotalSeconds)s"
Write-Host "Average Request Duration: $($metrics.AverageRequestDuration.TotalMilliseconds)ms"
Write-Host "Total Bytes: $($metrics.TotalBytesTransferred / 1KB)KB"

# Resource breakdown by type
$metrics.ResourceBreakdown | ForEach-Object {
    Write-Host "$($_.Key): $($_.Value) requests"
}

# Or get the full formatted report
Write-Host $metrics.GetReport()

Advanced Testing with Proxy

# Test through a proxy with authentication
$cred = Get-Credential
$result = Test-HtmlBrowser -Url 'https://example.com' `
    -Proxy 'http://proxy:8080' `
    -ProxyCredential $cred `
    -Timeout 60000

Batch Testing Multiple URLs

# Test multiple URLs and generate report
$urls = @(
    'https://example.com/home',
    'https://example.com/about',
    'https://example.com/contact'
)

$results = $urls | ForEach-Object {
    $result = Test-HtmlBrowser -Url $_
    [PSCustomObject]@{
        Url = $_
        Status = if ($result.Passed) { 'PASS' } else { 'FAIL' }
        LoadTime = $result.PageLoadTime.TotalSeconds
        Requests = $result.TotalRequests
        Failed = $result.FailedRequestCount
        Errors = $result.ErrorCount
        Warnings = $result.WarningCount
    }
}

# Display results in a table
$results | Format-Table -AutoSize

# Export to CSV for further analysis
$results | Export-Csv -Path 'browser-test-results.csv' -NoTypeInformation

# Find pages with issues
$results | Where-Object { $_.Status -eq 'FAIL' } | ForEach-Object {
    Write-Warning "Failed: $($_.Url) - $($_.Failed) failed requests, $($_.Errors) errors"
}

Integration with Pester Tests

# Save as MyWebsite.Tests.ps1
Describe "Website Browser Tests" {

    BeforeAll {
        $baseUrl = 'https://mywebsite.com'
    }

    It "Homepage should load without errors" {
        $result = Test-HtmlBrowser -Url $baseUrl
        $result.Passed | Should -BeTrue
        $result.ConsoleErrors.Count | Should -Be 0
        $result.FailedRequestCount | Should -Be 0
    }

    It "All CSS files should load successfully" {
        $result = Test-HtmlBrowser -Url $baseUrl
        $cssFiles = $result.CssResources

        $cssFiles.Count | Should -BeGreaterThan 0
        $cssFiles | ForEach-Object {
            $_.Status | Should -Be 200
            $_.ErrorType | Should -BeNullOrEmpty
        }
    }

    It "Page should load within 3 seconds" {
        $result = Test-HtmlBrowser -Url $baseUrl
        $result.PageLoadTime.TotalSeconds | Should -BeLessOrEqual 3
    }

    It "Console should not contain JavaScript errors" {
        $errors = Test-HtmlBrowser -Url $baseUrl -ErrorsOnly
        $errors | Should -BeNullOrEmpty
    }

    It "Total page size should be under 5MB" {
        $metrics = Test-HtmlBrowser -Url $baseUrl -PerformanceOnly
        $totalMB = $metrics.TotalBytesTransferred / 1MB
        $totalMB | Should -BeLessOrEqual 5
    }
}

# Run tests
Invoke-Pester -Path .\MyWebsite.Tests.ps1 -Output Detailed

Testing Local HTML Reports

# Test HTMLForgeX generated reports
$reportPath = "C:\Reports\MonthlyReport.html"

# Basic test
$result = Test-HtmlBrowser -Path $reportPath
if (-not $result.Passed) {
    Write-Warning "Report has issues:"
    $result.ConsoleErrors | ForEach-Object {
        Write-Warning "  JS Error: $($_.Text)"
    }
    $result.FailedRequests | ForEach-Object {
        Write-Warning "  Failed Resource: $($_.Url)"
    }
}

# Test multiple reports
Get-ChildItem -Path "C:\Reports" -Filter "*.html" | ForEach-Object {
    $result = Test-HtmlBrowser -Path $_.FullName
    [PSCustomObject]@{
        Report = $_.Name
        Status = if ($result.Passed) { '✅' } else { '❌' }
        LoadTime = "$($result.PageLoadTime.TotalSeconds)s"
        Errors = $result.ErrorCount
        MissingResources = $result.FailedRequestCount
    }
} | Format-Table -AutoSize

# Test with visible browser for debugging
$debugResult = Test-HtmlBrowser -Path $reportPath -Headless:$false -Timeout 60000

Monitoring and Alerting

# Monitor website health
function Test-WebsiteHealth {
    param(
        [string]$Url,
        [int]$MaxLoadTime = 5,
        [int]$MaxErrors = 0
    )

    $result = Test-HtmlBrowser -Url $Url

    $issues = @()

    if ($result.PageLoadTime.TotalSeconds -gt $MaxLoadTime) {
        $issues += "Slow load time: $($result.PageLoadTime.TotalSeconds)s"
    }

    if ($result.ErrorCount -gt $MaxErrors) {
        $issues += "Console errors: $($result.ErrorCount)"
    }

    if ($result.FailedRequestCount -gt 0) {
        $issues += "Failed requests: $($result.FailedRequestCount)"
    }

    if ($issues.Count -eq 0) {
        Write-Host "✅ $Url is healthy" -ForegroundColor Green
    } else {
        Write-Host "❌ $Url has issues:" -ForegroundColor Red
        $issues | ForEach-Object { Write-Host "   - $_" -ForegroundColor Yellow }

        # Send alert (example)
        # Send-MailMessage -To "admin@company.com" -Subject "Website Issue" -Body ($issues -join "`n")
    }

    return @{
        Url = $Url
        Healthy = $issues.Count -eq 0
        Issues = $issues
        Timestamp = Get-Date
    }
}

# Test multiple sites
$sites = @('https://site1.com', 'https://site2.com')
$healthChecks = $sites | ForEach-Object { Test-WebsiteHealth -Url $_ }

# Save results
$healthChecks | ConvertTo-Json | Out-File "health-check-$(Get-Date -Format 'yyyyMMdd-HHmmss').json"

C# Browser Testing

Basic Testing

using HtmlTinkerX;

// Run comprehensive test on URL
var result = await HtmlBrowserTester.TestUrlAsync("https://example.com");

// Test a local HTML file
var fileResult = await HtmlBrowserTester.TestFileAsync(@"C:\MyProject\index.html");

if (result.Passed)
{
    Console.WriteLine("✅ All tests passed!");
}
else
{
    Console.WriteLine($"❌ {result.Summary}");
}

// Analyze results
Console.WriteLine($"Total Requests: {result.TotalRequests}");
Console.WriteLine($"Failed: {result.FailedRequestCount}");
Console.WriteLine($"Errors: {result.ErrorCount}");
Console.WriteLine($"Warnings: {result.WarningCount}");

Testing Local HTML Files

// Test local HTML file with full analysis
var testResult = await HtmlBrowserTester.TestFileAsync(
    @"C:\Projects\MyReport\report.html",
    HtmlBrowserEngine.Chromium,
    headless: true,
    timeout: 30000);

// Check specific issues
if (testResult.ConsoleErrors.Any())
{
    Console.WriteLine($"Found {testResult.ErrorCount} JavaScript errors:");
    foreach (var error in testResult.ConsoleErrors)
    {
        Console.WriteLine($"  - {error.Text}");
        Console.WriteLine($"    Location: {error.FullLocation}");
        if (!string.IsNullOrEmpty(error.StackTrace))
        {
            Console.WriteLine($"    Stack: {error.StackTrace}");
        }
    }
}

// Analyze resource loading
var slowResources = testResult.NetworkEntries
    .Where(r => r.Duration > TimeSpan.FromSeconds(1))
    .OrderByDescending(r => r.Duration);

foreach (var resource in slowResources)
{
    Console.WriteLine($"Slow resource: {resource.Url} took {resource.Duration?.TotalSeconds}s");
}

Network Request Analysis

// Test and analyze network requests
var result = await HtmlBrowserTester.TestUrlAsync("https://example.com");

// Check CSS resources
foreach (var css in result.CssResources)
{
    Console.WriteLine($"CSS: {css.Url}");
    Console.WriteLine($"  Duration: {css.Duration?.TotalMilliseconds}ms");
    Console.WriteLine($"  Size: {css.TransferSize} bytes");
    Console.WriteLine($"  Cached: {css.ServedFromCache}");
}

// Check failed requests
foreach (var failed in result.FailedRequests)
{
    Console.WriteLine($"Failed: {failed.Url}");
    Console.WriteLine($"  Error: {failed.ErrorType} - {failed.ErrorMessage}");
}

// Check JavaScript resources
var jsFiles = result.JavaScriptResources;
var totalJsSize = jsFiles.Sum(js => js.TransferSize ?? 0);
Console.WriteLine($"Total JS size: {totalJsSize / 1024}KB");

Console Error Detection

// Get only console errors
var errors = await HtmlBrowserTester.TestConsoleErrorsAsync("https://example.com");

foreach (var error in errors)
{
    Console.WriteLine($"Error: {error.Text}");
    Console.WriteLine($"  Type: {error.Type}");
    Console.WriteLine($"  Location: {error.FullLocation}");
    Console.WriteLine($"  Timestamp: {error.Timestamp}");

    if (!string.IsNullOrEmpty(error.StackTrace))
    {
        Console.WriteLine($"  Stack: {error.StackTrace}");
    }
}

Performance Analysis

// Get performance metrics
var metrics = await HtmlBrowserTester.TestPerformanceAsync("https://example.com");

// Display performance report
Console.WriteLine(metrics.GetReport());

// Check specific thresholds
if (metrics.TotalLoadTime > TimeSpan.FromSeconds(5))
{
    Console.WriteLine("⚠️ Page load time exceeds 5 seconds!");
}

if (metrics.LongestRequest?.Duration > TimeSpan.FromSeconds(2))
{
    Console.WriteLine($"⚠️ Slow resource: {metrics.LongestRequest.Url}");
}

Testing Local HTML Files

// Test a local HTML file created by HTMLForgeX or other tools
var localResult = await HtmlBrowserTester.TestFileAsync(@"C:\Projects\MyReport\report.html");

// Check if all resources loaded correctly
if (localResult.Passed)
{
    Console.WriteLine("✅ Local HTML file passed all tests!");
}
else
{
    // Analyze what went wrong
    foreach (var failed in localResult.FailedRequests)
    {
        Console.WriteLine($"❌ Failed to load: {failed.Url}");
        Console.WriteLine($"   Error: {failed.ErrorType}");
    }
}

// Test with custom timeout for slow local resources
var slowResult = await HtmlBrowserTester.TestFileAsync(
    @"C:\MyProject\index.html",
    timeout: 30000  // 30 seconds
);

Integration Testing Examples

// Example: Testing in xUnit
[Fact]
public async Task Website_Should_Load_Without_Errors()
{
    var result = await HtmlBrowserTester.TestUrlAsync("https://mysite.com");

    Assert.True(result.Passed, $"Test failed: {result.Summary}");
    Assert.Empty(result.ConsoleErrors);
    Assert.Empty(result.FailedRequests);
    Assert.True(result.PageLoadTime < TimeSpan.FromSeconds(3),
        "Page load time exceeded 3 seconds");
}

// Example: Testing specific CSS resources
[Theory]
[InlineData("styles.css")]
[InlineData("theme.css")]
public async Task CSS_Files_Should_Load_Successfully(string cssFile)
{
    var css = await HtmlBrowserTester.TestCssResourceAsync(
        "https://mysite.com", cssFile);

    Assert.NotNull(css);
    Assert.Equal(200, css.Status);
    Assert.True(css.Duration < TimeSpan.FromSeconds(1));
}

// Example: Performance regression test
[Fact]
public async Task Page_Performance_Should_Meet_Thresholds()
{
    var metrics = await HtmlBrowserTester.TestPerformanceAsync("https://mysite.com");

    Assert.True(metrics.TotalLoadTime < TimeSpan.FromSeconds(5));
    Assert.True(metrics.TotalBytesTransferred < 5 * 1024 * 1024); // 5MB
    Assert.True(metrics.TotalRequests < 50);
    metrics.ResourceBreakdown.TryGetValue(
        HtmlNetworkResourceType.Image,
        out int imageRequestCount);
    Assert.True(imageRequestCount < 20,
        $"Image request count exceeds limit: {imageRequestCount}");
}

Batch Testing Multiple Pages

// Test multiple pages efficiently
var urls = new[] {
    "https://example.com/home",
    "https://example.com/about",
    "https://example.com/contact"
};

var results = await Task.WhenAll(
    urls.Select(url => HtmlBrowserTester.TestUrlAsync(url))
);

// Generate summary report
foreach (var (url, result) in urls.Zip(results))
{
    Console.WriteLine($"\n{url}:");
    Console.WriteLine($"  Status: {(result.Passed ? "PASS" : "FAIL")}");
    Console.WriteLine($"  Load Time: {(result.PageLoadTime?.TotalSeconds ?? 0):F2}s");
    Console.WriteLine($"  Requests: {result.TotalRequests} ({result.FailedRequestCount} failed)");
    Console.WriteLine($"  Console: {result.ErrorCount} errors, {result.WarningCount} warnings");
}

// Find slowest page
var slowest = results.OrderByDescending(r => r.PageLoadTime ?? TimeSpan.Zero).First();
Console.WriteLine($"\nSlowest page: {slowest.Url} ({(slowest.PageLoadTime?.TotalSeconds ?? 0):F2}s)");

Test Result Properties

HtmlBrowserTestResult

Url - The tested URL
PageLoadTime - Total page load duration
NetworkEntries - All network requests with detailed info
ConsoleEntries - All console messages
ConsoleErrors - Only error messages
ConsoleWarnings - Only warning messages
FailedRequests - Failed network requests
CssResources - CSS file requests
JavaScriptResources - JS file requests
ImageResources - Image requests
Passed - Whether all tests passed
Summary - Human-readable summary

HtmlNetworkEntryDetailed

Url - Request URL
Method - HTTP method
Status - Response status code
ProtocolVersion - HTTP protocol version
Duration - Request duration
ResourceType - Type of resource (Document, Stylesheet, Script, etc.)
TransferSize - Total bytes transferred
ServedFromCache - Whether served from cache
ErrorType - Error type if failed
ContentType - Response content type

HtmlConsoleEntryDetailed

Text - Console message text
Type - Message type (Error, Warning, Info, etc.)
Timestamp - When logged
SourceUrl - Source file URL
LineNumber - Line in source
StackTrace - Stack trace for errors
SeverityLevel - 1=Info, 2=Warning, 3=Error
IsError/IsWarning/IsInfo - Quick type checks

HtmlPerformanceMetrics

TotalLoadTime - Total time to load the page
TotalRequests - Number of network requests made
TotalBytesTransferred - Total bytes downloaded
AverageRequestDuration - Average time per request
LongestRequest - The slowest network request
ResourceBreakdown - Dictionary of requests grouped by type (Document, Stylesheet, Script, Image, Font, etc.)
GetReport() - Returns a formatted text report with all metrics

Playwright Auto-Setup

Playwright browsers are automatically downloaded on first use. No manual setup required. The download is cached per-user (default locations below).

How Auto-Download Works

When you first use browser testing, Playwright automatically downloads required components:

Playwright Driver & Node.js:
- Windows: %LOCALAPPDATA%\ms-playwright-driver
- macOS: ~/Library/Caches/ms-playwright-driver
- Linux: ~/.cache/ms-playwright-driver
- Contains the Playwright driver and embedded Node.js runtime
Browser Installations:
- Windows: %LOCALAPPDATA%\ms-playwright
- macOS: ~/Library/Caches/ms-playwright
- Linux: ~/.cache/ms-playwright
- Contains Chromium, Firefox, and/or WebKit browsers
Download Process:
- Shows progress: "Downloading Playwright driver... X% (Y MB/s)"
- Thread-safe - prevents concurrent downloads
- Subsequent runs use cached components - no re-download needed
- You can manually ensure Chromium is installed using HtmlBrowser.EnsureInstalledAsync(HtmlBrowserEngine.Chromium)

Linux: Avoiding sudo prompts

On Linux, Playwright can also install OS-level dependencies when invoked with --with-deps (this typically requires root/sudo).

By default, HtmlTinkerX only uses --with-deps when running as root to avoid unexpected sudo prompts during normal test execution. You can override this behavior by setting:

HTMLTINKERX_PLAYWRIGHT_WITH_DEPS=1 to force --with-deps
HTMLTINKERX_PLAYWRIGHT_WITH_DEPS=0 to never use --with-deps

Cleaning Playwright Cache

# View cache size and clean if needed
Clear-HtmlBrowserCache -WhatIf

# Force clean without confirmation
Clear-HtmlBrowserCache -Force

# Skip cleaning temporary files (only clean browser downloads)
Clear-HtmlBrowserCache -SkipTemp -Force

# Skip cleaning browser downloads (only clean temp files)
Clear-HtmlBrowserCache -SkipBrowsers -Force

# View detailed information about what will be cleaned
Clear-HtmlBrowserCache -Verbose

The enhanced cache cleaner now:

Cleans multiple Playwright cache locations (LocalAppData and .cache)
Removes temporary Playwright files from the temp directory
Cleans up trace files left behind by debugging sessions
Shows detailed size information for each location
Provides granular control over what to clean

C# Cache Cleaning

// Manually ensure browser is installed (usually not needed - happens automatically)
await HtmlBrowser.EnsureInstalledAsync(HtmlBrowserEngine.Chromium);

// Get all cache locations
var locations = HtmlBrowserCacheCleaner.GetCacheLocations();
Console.WriteLine($"Found {locations.Count} locations totaling {locations.Sum(l => l.SizeMB):F2} MB");

// Clean all cache
var result = HtmlBrowserCacheCleaner.CleanAllCache();
if (result.Success)
{
    Console.WriteLine($"Cleaned {result.TotalSizeClearedMB:F2} MB");
}
else
{
    Console.WriteLine($"Failed to clean {result.Failed.Count} locations");
}

// Clean only browser downloads
var browserResult = HtmlBrowserCacheCleaner.CleanAllCache(
    includeBrowsers: true,
    includeTemp: false);

// Get locations without cleaning (for inspection)
var tempOnly = HtmlBrowserCacheCleaner.GetCacheLocations(
    includeBrowsers: false,
    includeTemp: true);
foreach (var location in tempOnly)
{
    Console.WriteLine($"{location.Description}: {location.SizeMB:F2} MB at {location.Path}");
}

Integration with Test Frameworks

xUnit Example

[Fact]
public async Task WebsiteShouldHaveNoErrors()
{
    var result = await HtmlBrowserTester.TestUrlAsync("https://mysite.com");

    Assert.True(result.Passed, result.Summary);
    Assert.Empty(result.ConsoleErrors);
    Assert.Empty(result.FailedRequests);
}

[Fact]
public async Task CssShouldLoadQuickly()
{
    var result = await HtmlBrowserTester.TestUrlAsync("https://mysite.com");

    foreach (var css in result.CssResources)
    {
        Assert.True(css.Duration < TimeSpan.FromSeconds(2),
            $"CSS {css.Url} took {css.Duration?.TotalSeconds}s");
    }
}

Pester Example

Describe "Website Health Check" {
    It "Should have no console errors" {
        $result = Test-HtmlBrowser -Url "https://mysite.com"
        $result.ErrorCount | Should -Be 0
    }

    It "Should load all resources successfully" {
        $result = Test-HtmlBrowser -Url "https://mysite.com"
        $result.FailedRequestCount | Should -Be 0
    }

    It "Should load within 5 seconds" {
        $metrics = Test-HtmlBrowser -Url "https://mysite.com" -PerformanceOnly
        $metrics.TotalLoadTime.TotalSeconds | Should -BeLessThan 5
    }
}

🔧 Advanced Features

Browser Configuration

# Custom browser settings
$session = Start-HtmlBrowserSession -Url 'https://example.com' `
    -UserAgent 'Custom Bot 1.0' `
    -ViewportWidth 1920 `
    -ViewportHeight 1080 `
    -DeviceScaleFactor 2 `
    -Visible `
    -SlowMo 1000

Request Interception

# Mock API responses
$handler = Register-HtmlRoute -Session $session -Pattern '**/api/data' -ScriptBlock {
    param($route)
    Complete-HtmlRoute -Route $route -Options @{
        Status = 200
        ContentType = 'application/json'
        Body = '{"status": "success", "data": []}'
    }
}

# Navigate and test
Invoke-HtmlBrowserNavigation -Session $session -Url 'https://example.com/app'
Unregister-HtmlRoute -Session $session -Pattern '**/api/data' -Handler $handler

State Management

# Save browser state
Export-HtmlBrowserState -Session $session -Path 'session-state.json'

# Restore in new session
$newSession = Import-HtmlBrowserState -Path 'session-state.json' -Url 'https://example.com/dashboard'

🏗️ Third-Party Dependencies

HtmlTinkerX utilizes several high-quality open-source libraries:

📦 HTML & DOM Processing

AngleSharp - MIT License - Modern HTML5 parser
AngleSharp.Css - MIT License - CSS parsing and styling
AngleSharp.Js - MIT License - JavaScript engine integration
AngleSharp.Diffing - MIT License - HTML document comparison
Html Agility Pack - MIT License - Alternative HTML parser

🎨 Resource Optimization

NUglify - BSD 2-Clause License - HTML/CSS/JS minification
Jsbeautifier - MIT License - JavaScript formatting
PreMailer.Net - Apache 2.0 License - Email CSS inlining

🌐 Browser Automation

Microsoft.Playwright - Apache 2.0 License - Browser automation
ChartForgeX - MIT License - Dependency-free screenshot image composition

Screenshot image post-processing is routed through ChartForgeX. See Docs/Dependencies.md before changing that dependency path.

🔧 System Libraries

System.Net.Http - MIT License - HTTP client
System.IO.Compression - MIT License - Compression support
System.Threading.Channels - MIT License - Async communication

All dependencies are distributed under permissive licenses. Refer to each project's repository for complete license information.

📖 Documentation & Support

📚 Examples: Check the Examples folder for comprehensive usage samples
🐛 Issues: Report bugs and request features on GitHub Issues
💬 Discord: Join our Discord community for support and discussions
📝 Blog: Read detailed tutorials on evotec.xyz

🔄 Updates & Versioning

PowerShell Module Updates

Update-Module -Name PSParseHTML

NuGet Package Updates

dotnet add package HtmlTinkerX

⚠️ Important: Always test updates in a development environment before deploying to production. Breaking changes may occur between versions.

🔧 Troubleshooting

Browser Extraction Mode

Playwright browser missing: run Test-HtmlBrowser -Install or retry the command with -Clean when the browser cache is corrupt.
Corporate proxy or locked-down network: pass -Proxy and -ProxyCredential on URL-based commands, or use Set-HtmlHttpClientOption for reusable module defaults.
Timeouts on app shells: prefer -RenderProfile AppShell -WaitForSelector 'main' or Wait-HtmlBrowserReady -Stable instead of relying on network idle for applications that keep long-polling or WebSocket connections open.
Lazy content does not appear: use LazyLoadedContent, -AutoScroll, Invoke-HtmlBrowserScroll, or a specific -WaitForSelector that represents the content you actually need.
Login reuse fails: create storage state with the same browser engine and profile assumptions, then pass -StorageStatePath to the later render. Keep state files out of source control.
Response bodies contain secrets: use -IncludeResponseBody -RedactResponseBody, keep -ResponseBodyResourceType narrow, and lower -ResponseBodyMaxBytes for large APIs.
Static HTML is already enough: do not start a browser. Use Test-HtmlExtractionPlan, Find-HtmlDataSource, Invoke-HtmlDataExtraction, ConvertFrom-Html*, Find-HtmlApiEndpoint, crawl profiles, or Compare-HtmlStaticRendered to prove whether rendering adds value.
Endpoint extraction is skipped: Invoke-HtmlDataExtraction does not fetch endpoint sources unless -AllowHttpFetch is supplied, and high-risk endpoints remain blocked. Review RiskLevel, Warnings, IsExternal, and RequiresAuthenticationHint before opting into endpoint reads.
Diagnostics look noisy: Get-HtmlBrowserDiagnostics reports evidence such as navigator.webdriver, failed requests, console errors, and storage keys. Treat these as reliability hints, not a score to game.

Jint and JavaScript Parser Notes

Current builds use Jint 4.x and Acornima for JavaScript parsing. Older examples that reference Esprima or Jint 3.x types do not match the current API; use the Acornima types exposed by ConvertFrom-JavaScriptAst and Select-JavaScriptAstNode.

Browser Testing Issues

If browser tests fail:

First run downloads browsers automatically - This can take a few minutes (~400MB)
- You'll see: "Downloading Playwright driver... X% (Y MB/s)"
- This only happens once per system
Network timeout issues - Some sites may be slow or blocked
- Try increasing timeout: Test-HtmlBrowser -Url $url -Timeout 60000
- Test with a simple URL first: Test-HtmlBrowser -Url "http://httpbin.org/html"

Behind a proxy - Set proxy environment variables:

$env:HTTPS_PROXY = "http://proxy:8080"
$env:HTTP_PROXY = "http://proxy:8080"

Or use proxy parameters:

Test-HtmlBrowser -Url $url -Proxy "http://proxy:8080" -ProxyCredential (Get-Credential)

Clean and retry if you suspect corrupted downloads:

Clear-HtmlBrowserCache -Force
# Then run your test again - it will re-download browsers

Manual browser installation (C#):

// Ensure browser is installed before testing
await HtmlBrowser.EnsureInstalledAsync(HtmlBrowserEngine.Chromium);

📄 License

HtmlTinkerX and PSParseHTML are available under the MIT License. Third-party components remain under their own license terms.