< Previous Challenge - Home - Next Challenge >

---

Challenge 07 - Image Generation using DALL-E

Introduction

Now it's time to introduce Image generation to the reference application using DALL-E. DALL-E is an artificial intelligence (AI) model that generates images from textual descriptions. DALL-E can create images of objects, scenes, and even abstract concepts based on the descriptive text provided to it. This capability allows for a wide range of creative possibilities, from illustrating ideas to creating entirely new visual concepts that might not exist in the real world.

Description

In this challenge, you will deploy an Azure AI Foundry service capable of hosting DALL-E models and integrate it with the Semantic Kernel. You will also create a plugin to generate images using DALL-E from a text prompt.

Challenges

1. Create an Azure AI Foundry Deployment for DALL-E in a region capable of hosting DALL-E models.

2. Update the reference application by adding the DALL-E model to Semantic Kernel

   NOTE: The Semantic Kernel Documentation In-Depth Samples provides examples of using Text-to-Image models like DALL-E. Be sure to modify the sample to use an Azure AI Foundry model instead of an OpenAI.

3. Create a Semantic Kernel plugin to generate an image using DALL-E from a text prompt. The plugin should accept a text prompt and return the URL string for the image generated by DALL-E.

4. A simple prompt to test the plugin

   create a picture of a cute kitten wearing a hat
   

5. Working with chat history to generate images

   :exclamation: Refresh browser to clear chat history before entering the next prompt

   NOTE: Feel free to change the details of the story to make it your own.

   Generate a detailed children's story about a dragon and a little girl that go on an adventure together
   

   :x: Without clearing the chat history, create an image from a scene in the story.

   randomly choose a major scene from the story and create a cartoon style image
   

   :bulb: Set a breakpoint in the image plugin to view the generated prompt sent to the DALL-E model. Notice how the LLM summarized a scene from the story to generate a prompt for the text-to-image model.

   :exclamation:Refresh browser to clear chat history before entering the next set of prompts

6. Write a prompt to call multiple plugins.

   Create a prompt that calls the image plugin and at least one other plugin written in the previous challenges. Try to use as many plugins as you can in a single prompt.

7. Finally, Let's do some product design.

   NOTE: Feel free to change the details of the product

   In this final task, have the AI generate a product name, description and an image for a handheld teleporting device using a single prompt. This will require the AI to construct a multi-step plan that will:

   1. Generate a product name 
   2. Generate a product description
   3. Create a prompt from the name and description suitable for a text-to-image AI model
   4. Call the image plugin with the generated prompt
   5. Generate a prompt that will create a logo for the product
   6. Call the image plugin again with the Logo prompt
   

   :bulb: Set a breakpoint in the image plugin to view the generated prompt sent to the DALL-E model. Notice how the LLM summarized the product name and description to generate a prompt for the text-to-image model.

Success Criteria

1. Verify that your Image plugin can generate images from simple text prompts.
2. Verify that your Image plugin can work with chat history to generate relevant images.
3. Verify that your Image plugin can be called from a prompt that also calls other plugins.

Learning Resources

Create and deploy an Azure AI Foundry Service resource

Semantic Kernel Samples

Add native code as a plugin to Semantic Kernel

< Previous Challenge - Home - Next Challenge >