Microsoft Build 2023: Strengthening Developers via Microsoft 365 Copilot Plug-ins

Microsoft’s famous Build conference, held from May 23 to 25 in Seattle, Washington, brought exciting announcements for developers. This year’s focus was on artificial intelligence (AI) with announcements about innovative AI projects and future plans.

The first day saw some big reveals, including the collaboration between Microsoft and OpenAI to enrich and grow the AI plug-ins ecosystem. Microsoft has chosen an open standard for plug-ins that integrate seamlessly with OpenAI’s ChatGPT and Microsoft’s Copilot solutions such as Bing Chat, Dynamics 365 Copilot, Windows Copilot and Microsoft 365 Copilot. These are exciting times for AI and plug-in development!

Microsoft 365 Copilot – The future of natural language

There is a plethora of plug-ins available to Microsoft 365 Copilot users. These include Microsoft Power Platform connectors, Teams message extensions and ChatGPT plug-ins. The plug-ins are designed to make your life as a developer easier by integrating with your existing software, tools, knowledge and skills. There is always a plug-in waiting for you to improve your development skills, whether you prefer ChatGPT, Teams or the Power Platform.

Effortless plug-in creation with Teams Toolkit

Developers can use the Teams Toolkit for Visual Studio, Visual Studio Code and CLI to create Teams message extensions that work like Microsoft 365 Copilot plug-ins. With added functionality in the Teams Toolkit, Microsoft aims to simplify the process of developing, testing and debugging plug-ins. Developers can easily add any API of the OpenAPI specification to Microsoft 365 Copilot through the plug-in creation experience of the Teams Toolkit. This functionality is currently available in a private preview.

Customizing the user experience with adaptive maps

By using Adaptive Cards, developers can have complete control over user interface customization when their plug-in is used. Based on the metadata in the OpenAPI specification, the Teams Toolkit creates a plug-in consisting of a manifest and declarative Adaptive Cards, which define the Copilot user experience. Before testing their plug-in in Copilot, developers can further customize the manifest and maps to their specific scenario, including specifying login settings.

Enrich your data with the semantic index for Copilot

Microsoft Graph provides access to customer data related to compliance, security, privacy, productivity and collaboration. Copilot uses Microsoft Graph data to generate user prompts and responses, while providing permissions and security at runtime. Importantly, this data remains solely within the client’s tenants and is not used to train Microsoft’s base models.

Introduction of the semantic index for Copilot

The semantic index for Copilot is a complex map of user and business data that enables Copilot to display Microsoft Graph data. The semantic index captures semantics and relationships around content and users by constructing vector inclusions based on Microsoft Graph data. This allows Copilot to quickly search through billions of objects in a vector index using semantic search. To use the semantic index for Copilot, developers can use Graph connectors to upload their data to Microsoft Graph and receive personalized and useful results.

Access enterprise data with Microsoft Dynamics 365 and Power Platform Microsoft 365

Copilot now provides access to structured data from Microsoft Dynamics 365 and Microsoft Power Platform stored in Microsoft Dataverse, in addition to productivity and collaboration data in Microsoft Graph. Customers in the Microsoft 365 Copilot Early Access program will have the opportunity to test out this feature as Microsoft 365 Copilot is rolled out.

Developers shape the future of work with Microsoft 365 Copilot

As customers start using Microsoft 365 Copilot with connected apps and services, developers will be crucial in establishing new work patterns. In a simulation scenario, this potential is illustrated by a user at Dentsu Inc. using Microsoft 365 Copilot in conjunction with Atlassian’s Jira plug-ins and a line-of-business Dentsu app plug-in, and data from Atlassian’s Confluence integrated via a Microsoft Graph connector.

Accelerate success for any developer

With the new extension architecture for Microsoft 365 Copilot, developers can leverage their AI skills and reach millions of Microsoft 365 users. With comprehensive support for the entire app lifecycle, including tooling, distribution, management, commerce and enterprise readiness, developers can create plug-ins for Copilot with their existing knowledge, code and resources.

More innovations to improve app development

In addition to the extensibility of Copilot, there are other exciting developments to improve app development. Thanks to the Live Share SDK, Teams meetings can now collaborate in real time. Microsoft Teams’ configurable avatars and reactions and immersive spaces, both currently in private preview, increase engagement and transcend physical boundaries. Microsoft Mesh, also currently in private preview, allows developers to create unique, immersive experiences that promote connection and cohesion, no matter where users are physically located.

Explore the future of work with Microsoft 365 Copilot

If Microsoft 365 Copilot participates in the Early Access program, customers will soon be able to test the expansion features announced at Microsoft Build. By preparing for the release of Copilot, developers and partners can use all available tools and resources to get the most out of Microsoft 365 Copilot.

Strengthening AI capabilities with ONNX Runtime and Olive

The Hybrid Loop development model introduced last year has enabled hybrid AI solutions for Azure and client devices. We are pleased to announce that this vision has been realized using Olive, a toolchain that facilitates optimizing models across Windows and other devices, and ONNX Runtime, the gateway to Windows AI. ONNX Runtime enables the execution of AI models on Windows and other devices with CPU, GPU, NPU or a combination with Azure. These are the same tools we use internally.

ONNX Runtime and Azure EP Preview for Hybrid Inference ONNX

Runtime now supports the same API for running models locally or in the cloud. This allows you to leverage local resources when they are available and cloud resources when they are needed in hybrid inference scenarios for your app. You can now connect to models in AzureML or the Azure OpenAI service with the new Azure EP preview. By defining the cloud endpoint and cloud usage, you can gain greater control over cost and user experience.

Optimizing AI models with Olive and ONNX Runtime

Olive, an extensible toolchain, provides advanced model compression, optimization and compilation methods to optimize your models for different hardware targets. With ONNX Runtime, you can extend your Windows AI investment to all your app platforms, including Windows, iOS, Android and Linux. Olive and ONNX Runtime both accelerate the implementation of your AI models in applications, making it easier to build great AI experiences with less technical effort and better performance.

Partner Success Stories Using NPUs on Windows

Partners such as WhatsApp, Luminar Neo and Camo are seeing significant performance improvements and unlocking new user experiences by using NPUs to run their AI models. Deliver AI experiences like Camo with your apps by integrating the ONNX Runtime into the Windows platform. We look forward to seeing what the Windows developer community will create with these new AI toolchains and hardware upgrades for the NPU.

Productivity improvements with Dev Home

We discuss the new features and improvements Windows 11 offers to increase developer productivity. We understand that a developer’s workflow can be interrupted by various manual activities. That’s why we’re introducing Dev Home, a new productivity partner for Windows 11 developers. By using Dev Home, you can get back into the flow and organize your work.

One of Dev Home’s features is WinGet configuration, which streamlines and speeds up the machine installation procedure. With a WinGet configuration file, you can avoid the time-consuming manual process of setting up a new system or starting a project. All you need to do is run one command to start coding!

Dev Drive

A customized storage volume for developers We also introduce Dev Drive, a storage volume designed specifically for developers. The Resilient File System serves as the foundation and provides enhanced performance and security. To make managing storage locations with thousands of files and folders easier, Dev Drive has been enhanced to store project source code, working folders and package caches.

Customizable dashboard and GitHub integration in Dev Home

Dev Home comes with a customizable dashboard that allows you to track all your workflows and tasks in one place. You can even add GitHub widgets to effectively manage code-assignments, pull-requests and projects. In partnership with Team Xbox, we are bringing the GDK (Game Development Kit) to Dev Home. You can even add your own custom extensions to further personalize your experience.

Windows Terminal gets Smarter with GitHub Copilot integration

Thanks to GitHub Copilot’ s integration, your favorite tool, Windows Terminal, now becomes even smarter. The Terminal application uses natural language AI to suggest commands, describe errors and assist you in completing tasks. We are also testing GitHub Copilot in other development tools, such as WinDBG.

Taskbar Improvements for a Better Workflow

Based on your feedback, we have made changes to the taskbar. You can now easily identify each instance of an app stored in the taskbar and open it with just one click. The app instances are all labeled individually. In addition, you can hide the time and date in the taskbar to record your screen without removing the time and date so you can stay focused. You can also terminate a program without opening Task Manager by simply right-clicking on the application in the taskbar.

These are just some of the fascinating new features and improvements Windows 11 offers developers. We are determined to lighten the load and bring the fun and joy back into development. Stay tuned for further updates, we can’t wait to see what you do with these new features!

Updates and Improvements in the Microsoft Store on Windows

The Microsoft Store for Windows also got some interesting news from Panos! We update the Microsoft Store platform because we want to give you the best tools and opportunities to create excellent apps. Here’s what’s new:

Microsoft Store AI hub: The best of AI-powered experiences created by Microsoft and the developer community will be showcased in a unique section of the Microsoft Store to be unveiled soon. It will be a carefully curated selection of top AI apps and materials.

AI-Generated Keywords: We are releasing a handy tool in the Partner Center that allows you to generate search tags for your apps using AI. This allows you to improve app detection and make it easier for customers to find your app in the Store.

AI Generated Review Summary: We know that reading through a large number of app reviews can be time consuming. To provide users with a quick and easy way to browse app reviews, we use the power of AI to collect and summarize reviews. This allows users to find new information faster.

Microsoft Store Ads: Starting in June, we are expanding Microsoft Store Ads to more than 150 regions worldwide, increasing reach outside the United States. A new premium spotlight placement we are introducing gives you more opportunities to reach your target audience with high visibility in Store and Bing.com search results.

Backup Apps and Restore: We understand how important it is for you to retain customers when they switch devices. That’s why we are improving app backup and recovery features. You can test this update and provide feedback as it is currently in preview and accessible to Windows Insiders.

Conclusion

The latest improvements and changes to the Microsoft Store for Windows give developers more control than ever before. Thanks to new AI-powered features, enhanced advertising capabilities and improved user outreach tools, the potential for developing and showcasing excellent apps is greater than ever.

The launch of the Microsoft Store AI Hub will curate the best AI experiences, and app discovery will be accelerated by AI-generated review summaries and keywords. In addition, a global audience is now available thanks to the expansion of Microsoft Store Ads to more than 150 regions worldwide.

With enhanced backup and recovery capabilities, developers can confidently deliver great experiences and keep customers on multiple devices. These updates open a fascinating new chapter in the development of Windows, and we can’t wait to see the amazing creations and performance they will spur.

Source: Microsoft