README.md

Introduction

This section explores a system architecture for an agentic agent system that can be utilized across various agent scenarios and design patterns. The architecture encompasses software component-level design and addresses engineering considerations such as integration, scalability, security, logging, monitoring, and responsible AI.

Agent System Architecture: Logical View

The logical view offers an overview of the agentic system in terms of its functional building blocks. These components are elaborated upon in subsequent views detailing their construction. The architecture's primary services include:

• Hosted Foundation Models: These are the key foundation models (LLM/VLM) that drive the core capabilities of the agent.

• AI Safety Services: These services monitor content generated by both foundation models and humans, detecting and mitigating potential risks, threats, and quality issues.

• AI Services: Off-the-shelf AI services that provide additional capabilities for the agents, such as text-to-speech, speech-to-text, and object detection. These services might be needed to add capabilities that foundation models do not have (e.g., GPT-4 does not have audio capability) or are not robust or cost-effective to use (e.g., a single-purpose text classification model can be more robust and faster than a prompted LLM).

• Hosted Custom AI/ML Models: Custom machine learning and AI models that offer supplementary capabilities for the agents in scenarios where standard AI services are insufficient. For example, a custom intent classification model trained on a proprietary dataset to ensure fast and accurate intent classification.

• External Systems: These are external applications and systems that agents interact with to retrieve information or execute business transactions. The interaction with external systems is facilitated through APIs with appropriate security and access controls. These systems can range from publicly accessible tools like Bing and Google search services to enterprise applications like CRM systems.

• Agent Orchestration Framework: This central component is where the primary capabilities of the agents are developed. It includes:

  • Policy: Defines the rules and guidelines that govern agent behavior.
  • Memory: Manages the storage and retrieval of information that agents use in their reasoning processes.
  • Tool Calling: Facilitates the invocation of external tools and services that the agent might need to perform specific tasks.
  • Code Execution Environment: Provides a safe and controlled environment for executing code that the agent generates.
  • Multi-Agent Orchestration: Coordinates the interactions and collaborations between multiple agents to achieve complex goals.

• User Experience (Text, Voice, Video): This component handles the different modalities through which users interact with the agents, ensuring seamless user experiences across text, voice, and video interfaces.

• Application Platform: This layer provides the foundation for building and running agent applications. It includes:

  • Front End: The user interface that users interact with.
  • Backend: The server-side logic that powers the agent applications.
  • Database: The storage system for persisting data used by the agents and applications.

• Infrastructure Platform: The underlying hardware and software infrastructure that supports the entire system, ensuring scalability, reliability, and performance.

Agent System Architecture: Component View

The component view delves deeper into the technical components of each logical building block outlined in the previous section. This detailed architecture provides a clearer picture of the specific technologies and services used to implement the agent system.

Cloud AI Services

• Hosted Foundation Models:

  • Azure AI Studio/Azure ML: Options from Model Catalog such as models from OpenAI, Mistral AI, Meta. These are Large language models (LLMs) or Vision language models (VLMs) that power the core capabilities of the agent. These models can be provisioned as serverless APIs or they can be provisioned with dedicated managed computes.
  • Assistant API: OpenAI's Assistant APIs provide built-in agent capability such as stateful execution, code execution, file indexing (memory) so this can be a great option to simplify the development of agent system.

• Custom ML Models:

  • Azure Machine Learning: AML Platform can be used to train and deploy custom machine learning models tailored to specific needs.
  • AI Studio: AI Studio is a great option for no code/low code custom ML development and deployment.

  Note the use of GenAI foundation models such as GPT-4 to generate high quality labeled data using instructions and examples to accelerate custom ML model development regardless of tools used.

• Responsible AI:

  • Jailbreaking and Misuse: Bypassing safety restrictions can lead to generating harmful or inappropriate content.
  • Content Safety: GenAI may inadvertently produce violent, explicit, or otherwise harmful material.
  • Harmful Content Generation: GenAI can create realistic fake content (e.g., deepfakes, fake news) for malicious purposes like misinformation and fraud.
  • Bias and Discrimination: GenAI may perpetuate biases present in training data, causing discriminatory outcomes.
  • Privacy Violations: Improper management can lead to revealing personal or sensitive information.
  • Cybersecurity Threats: GenAI can be used for cyber attacks, such as creating phishing emails or malicious software.

  Azure AI content Safety API provides a wide range of ever-growing services to address the mentioned risks over text and images data. Sometimes it might be necessary to customize the detection model with custom labeled data using Content Safety Studio or custom ML model development.

• Azure AI Services:

  • Azure AI Services provides a suite of pre-built AI services for various capabilities, including text, vision, audio and other AI functionalities.
  • Example of how to use: Speech to text and text to speech can be used to enable voice capabilty in agent, a fast text classification model can be used to detect certain intent for routing to the right agent in multi-agent scenario and a high quality object detection model from Azure AI vision can be used as a tool to help Agent locate object's location accurately which current generation of VLM is not good at.

External Systems

• Internet Services:

  • Bing, Google, SERPAPI: External internet services for web search and information retrieval. These APIs are useful for enabling the agent to answer real-time general knowledge questions by searching the web.

• Enterprise Applications:

  • CRM, Finance: Internal enterprise applications such as customer relationship management (CRM) and financial operations (ERP). These applications provide the data that the agent needs to look up information or to provide an environment to execute business transactions.

• Enterprise DB:

  • In certain scenarios, such as analytics, agents may need to retrieve data directly from DBMS or even run complex queries against the database. Giving access to the DBMS for agents to execute against needs to be done with proper security safeguards such as role-based access control and row-level security.

Agent Services

• Code Execution Service:

  • App Services, Kubernetes Services, Function Apps: Platforms for executing code generated by the agent in a secure and scalable manner. The code execution environment should be an isolated sandbox environment with well-defined privileges.

• Agent Orchestration Service:

  • Langchain, Autogen, SK, Promptflow: Tools and frameworks for orchestrating agent behaviors and interactions.
  • App Services, Kubernetes Services, Function Apps: Platforms supporting the orchestration service.

• Current Memory:

  • Cache Redis, Azure Cosmos DB: Systems for managing short-term memory and fast data access. This is typically the memory of the current conversation with the user that is valid within a session. Note that the memory may be more than just textual conversation history but may also include complex objects such as results of code execution/tool calls, tool call history, and data retrieved from external systems.

• Long Term Memory:

  • Storage Accounts, AI Search: Systems for storing and retrieving long-term data and knowledge. This is normally a vector database that stores information that is indexed with embeddings of content and other metadata information so that the agent can retrieve detailed content when required. Storage Accounts can be used to store rich format data such as video and images.

Application Services

• App Backend Services:

  • App Services, Kubernetes Services, Function Apps: Backend components that run core application logic, including interacting with agent services. It can be implemented with frameworks such as FastAPI for Python, Node.js for JavaScript, etc.

• App Frontend Services:

  • App Services, Kubernetes Services, Function Apps: Frontend components to support rendering front-end logic and UI experience. It can be implemented with technologies such as JavaScript, REACT, etc.

• App Databases:

  • Azure Cosmos DB, Azure SQL, Cache Redis: Databases for storing application configuration data.

User Experience

• Client Applications:

  • Windows, macOS, iOS, Android, Web: Platforms for user-facing applications.
  • Text, Voice, Vision: Various modalities for user interaction.

• Small Language Model:

  • Lightweight language models for specific tasks and localized processing.

Supporting Services

• API Management:

  • Centralized management of APIs for secure and efficient communication between components.

• Network Security and Access Management:

  • Ensures secure access and communication across the network.

• Logging and Monitoring:

  • Tools and services for tracking system performance, detecting issues, and ensuring reliability.

Agent System Architecture: Infrastructure View

The infrastructure view provides a detailed look at the technical infrastructure components supporting the agent system. This view highlights the various resources and services used to ensure the system's scalability, reliability, and security.

Workload Resources

• Application Gateway + Web Application Firewall:

  • Zone-redundant: Ensures high availability and protection from web vulnerabilities.

• App Services:

  • Integration Subnet: Private endpoints for secure communication.
  • Multiple instances across Zone 1, Zone 2, and Zone 3 for load balancing and redundancy.

• Networking:

  • Private endpoints: Secure access to resources.
  • Subnets: Segregate and secure different network segments.
  • Network Security Groups: Control inbound and outbound traffic to network resources.

• Managed Identity:

  • Provides secure identity management for accessing Azure resources.

• Build Agents / Jump Boxes:

  • Dedicated resources for building and deploying applications.

Cloud Services

• Azure Cosmos DB:

  • Globally distributed, multi-model database service.

• Redis Cache:

  • In-memory data structure store used as a cache and message broker.

• Azure Key Vault:

  • Securely stores and accesses secrets, keys, and certificates.

• Azure Machine Learning:

  • Comprehensive machine learning service for building and deploying models.

• Azure Storage:

  • Scalable cloud storage for data, files, and applications.

• Azure AI Search:

  • AI-powered cloud search service for building sophisticated search experiences.

• Azure OpenAI Service:

  • Provides access to OpenAI's powerful language models.

• Azure Container Registry:

  • Stores and manages container images for Kubernetes deployments.

Monitoring

• Log Analytics Workspace:

  • Collects and analyzes log data from various sources.

• Diagnostic Settings:

  • Configures and collects diagnostic data for monitoring and troubleshooting.

Subscription Vending Provisioned Resources

• Management Group Placement:

  • Organizes and manages subscriptions within a management group.

• Spoke Virtual Network:

  • DNS provided by the hub for network segmentation and security.

• User Defined Route:

  • Custom routes for directing network traffic.

• Cost Management:

  • Tools for monitoring and managing cloud costs.

• Defender for Cloud Enrollment:

  • Security management and threat protection for cloud resources.

• Org Policy Assignments and Role Assignments:

  • Ensures compliance and assigns appropriate roles and permissions.

Connectivity Subscription

• Hub Virtual Network Region:

  • Centralized network hub for managing connectivity between resources.

• Azure Firewall and Azure Bastion:

  • Provides secure access to virtual machines and resources.

• VPN Gateway or ExpressRoute:

  • Establishes secure connections between on-premises networks and Azure.

• Private DNS Zones for Private Link:

  • Facilitates private DNS resolution for Azure services.

• Azure DNS Private Resolver:

  • Resolves DNS queries within a virtual network.

• Azure DDoS Protection:

  • Protects applications from Distributed Denial of Service (DDoS) attacks.

• Network Watcher:

  • Monitors and diagnoses network conditions.

• Policy Assignments and Role Assignments:

  • Enforces policies and assigns roles to ensure security and compliance.