May as been a staggering month for Microsoft and AI. I try in this long blog post to recap what’s really available or at least available in preview today. The post is organized in 3 sections:
- Azure AI including OpenAI news and improvements
- Azure AI and Azure platform services news (all related to AI Enhancements)
- Microsoft Copilots advancements cross the board
Microsoft has made significant advancements in AI, including Azure AI, Azure platform services, and Microsoft Copilots. These advancements offer more affordable models and solutions, such as Models as a Service (MaaS), which simplifies the development of generative AI applications without the need for dedicated infrastructure. Microsoft is also committed to responsible AI practices, with new features in Azure AI Content Safety, such as Custom Categories, Prompt Shields, and Groundeness Detection, which provide a precise and relevant approach to content safety across diverse platforms. Adopting AI offers great benefits, including faster AI app prototyping, responsible generative AI development, and seamless data integration
Azure AI News
The introduction of Custom Generative AI, a novel model type entering preview, promises to change the way Gen AI applications based on documents are written. This innovation allows users to initiate model creation with a single document, streamlining the schema definition and model creation process with minimal labeling. Leveraging large language models (LLMs), this feature enables users to extract fields and refine outputs, adapting seamlessly to new training data for continuous improvement post-deployment.
Azure AI Search has also seen significant enhancements, notably a dramatic increase in storage capacity and vector index size, provided at no additional cost. These upgrades support scaling generative AI applications efficiently. Notable updates include the ability to perform Retrieval-Augmented Generation (RAG) at scale with enhanced capacity and new features to support binary vector types, improving storage efficiency and response accuracy. Additionally, the integration of image vectorization via Azure AI Vision and other advanced models facilitates seamless data processing, allowing native image searches alongside text embeddings.
The integration of Azure AI Search with OneLake expands data indexing capabilities, enabling organizations to connect their data in Microsoft Fabric directly to Azure AI Search. This broadens the spectrum of data sources that can be indexed and searched, enhancing data accessibility and utility.
Microsoft Azure AI Studio, now generally available, brings together fast AI app prototyping, and responsible generative AI development. It supports complex applications ranging from content generation to project management through a dual development approach that combines a user-friendly interface with code-first capabilities. This platform allows developers to orchestrate multiple APIs and models, ground them on protected data, and conduct comprehensive performance and safety evaluations, ensuring scalable and continuously monitored AI solutions.
Additional features in Azure AI Studio include code-first development experiences with Azure Developer CLI (azd) and an AI Toolkit for Microsoft Visual Studio Code, which facilitate large language model operations as part of CI/CD solutions. This integration accelerates code-to-cloud workflows and enables local and cloud compute for fine-tuning models, currently available in preview.
Developers now have access to the latest foundation models from leading innovators through Models as a Service (MaaS), simplifying the development of generative AI applications without the need for dedicated infrastructure. This includes models from AI21, Bria AI, and Stability AI, with additional integrations from Arize and ClearML for preferred development tools.
The comprehensive AI toolchain in Azure AI Studio supports seamless data integration, prompt orchestration, and system evaluation. The preview of prompt flow enables workflow orchestration for multimodal models, including the use of images in conversations. Enhanced tracing and debugging features provide deeper insights into AI workflows, while monitoring tools help organizations track key metrics and ensure continuous improvement in generative AI applications.
At the forefront of the latest AI advancements announced at Microsoft Build, OpenAI’s flagship model, GPT-4o, is now generally available on Azure AI Studio and as an API. Faster, cheaper, multi-modal… what else?
Well, maybe the refreshed Assistants v2 API, now publicly available, that offers several significant updates. These include a file search tool, vector storage, and parameters for managing token usage such as max completion and max prompt token support. Users can now specify tools with the tool_choice parameter and create custom conversation histories using the assistant role in Threads. Additionally, support for temperature, top_p, and response_format parameters has been added. The API also includes streaming and polling support via the Python SDK, allowing for efficient response streaming and object status updates without the need for continuous polling.
The platform also previews a global standard deployment type, leveraging Azure’s global infrastructure to optimize traffic routing and provide the highest default quota for new models without requiring load balancing across multiple resources. Fine-tuning updates for GPT-4, now in public preview, include support for seed, events, full validation statistics, and checkpoints.
Another notable update is the preview of messaging insights for WhatsApp, facilitated by the Azure OpenAI Service via Azure Communication Services. This new feature will enable businesses to extract and utilize meaningful insights from WhatsApp messages, enhancing their communication strategies.
Microsoft also introduced Phi-3-vision, a new multimodal model in the Phi-3 family of AI small language models (SLMs). Phi-3-vision, sized at 4.2 billion parameters, excels in general visual reasoning tasks, including chart, graph, and table reasoning. It allows for the input of images and text to generate text responses, making it versatile for various applications. Alongside this, Phi-3-mini and Phi-3-medium are now generally available as part of Microsoft Azure AI’s Model as a Service (MaaS) offering, with Phi-3-small also joining the lineup.
Microsoft Azure AI is also reinforcing its commitment to responsible AI practices with new features in Azure AI Content Safety. Key enhancements include the forthcoming introduction of Custom Categories, empowering users to create bespoke filters for generative AI applications. These filters can be tailored to specific content safety needs or responsible AI policies, providing a precise and relevant approach to content safety across diverse platforms. Custom Categories will also offer rapid deployment options, allowing users to address incidents and emerging threats swiftly by implementing new filters in less than an hour. Furthermore, the introduction of Prompt Shields and Groundedness Detection, currently in preview, in Microsoft Azure OpenAI Service and AI Studio, underscores Azure AI’s dedication to enhancing safety for large language models (LLMs). These features are critical in mitigating both indirect and jailbreak prompt injection attacks and in detecting when LLMs produce ungrounded or hallucinated outputs.
Additionally, configurable content filters are now available for DALL-E 2 and 3 and GPT-4 Turbo Vision GA deployments, enhancing content moderation capabilities. Asynchronous filters are now available to all Azure OpenAI customers, improving latency in streaming scenarios. Lastly, Prompt Shields offer robust protection against direct and indirect attacks on applications powered by Azure OpenAI models, ensuring secure and reliable AI operations. These comprehensive updates underline Azure’s commitment to providing cutting-edge AI capabilities and secure, scalable solutions for developers and enterprises alike.
Microsoft Azure AI Speech has unveiled several innovative features designed to help developers create high-quality, voice-enabled applications. These updates, currently in preview and gated, introduce robust tools to enhance audio and video data processing. One of the key features is Speech Analytics, a service that automates the entire workflow for enterprises seeking to extract valuable insights from audio and video content. By integrating capabilities such as transcription, summarization, speech recognition, speaker diarization, and sentiment analysis, Speech Analytics facilitates the extraction of critical information from various audio and video sources, including customer feedback, call center recordings, podcasts, interviews, and more.
Another notable addition is the Video Dubbing service, which enables developers to translate video files into multiple supported languages, thereby reaching global audiences with high-quality translations. This service allows users to upload single or multiple videos for automatic translation and generation of video content in different languages. By simplifying the video dubbing process, developers can build their own dubbing pipelines with a single click, ensuring efficient and cost-effective delivery of multilingual video content.
AI in Azure platform Services
The integration of AI into the Microsoft Azure platform services was a focal point of the recent announcements at Microsoft Build, showcasing how AI is becoming ubiquitous across the ecosystem. One of the key highlights is the introduction of Real-Time Intelligence within Microsoft Fabric, an end-to-end software as a service (SaaS) solution currently in preview. This service empowers users to act on high-volume, time-sensitive, and granular data in a proactive manner, enabling faster and more informed business decisions. Real-Time Intelligence is designed to cater to a wide range of users, from everyday analysts using low-code/no-code interfaces to professional developers with code-rich user experiences. It features AI-powered insights integrated with Microsoft Copilot for generating queries and a one-click anomaly detection experience, allowing users to detect unknown conditions with high granularity in large datasets.
Azure Database for PostgreSQL has been enhanced with an Azure AI extension, allowing developers to leverage large language models (LLMs) to build sophisticated generative AI applications. Additionally, the in-database embedding generation feature, now in preview, enables AI models to understand relationships and similarities within data directly inside the database. This feature helps reduce embedding creation time to single-digit millisecond latency, ensures predictable costs, and maintains data compliance for confidential workloads.
AI skills in Fabric, designed to weave generative AI into specific data work within Fabric, empower users of all technical levels to build intuitive AI experiences. With AI skills in Fabric, users can ask questions and receive insights as if consulting an expert colleague while respecting security permissions. This feature is in preview and aims to unlock deeper insights from data. Furthermore, Copilot in Fabric, available generally in Power BI and in preview in other workloads, leverages the Azure OpenAI Service to help customers fully utilize their data, creating dataflows, pipelines, machine learning models, and visualizations through conversational language.
Microsoft Azure Cosmos DB, known for its AI capabilities, has introduced built-in vector database functionalities. This update, now in preview, includes vector indexing and vector similarity search, eliminating the need for separate vector databases and ensuring data and vectors remain synchronized. This enhancement, powered by DiskANN, promises highly performant and accurate vector search at any scale.
The AI Toolkit for Visual Studio Code, currently in preview, equips AI engineers with essential tools to develop and deploy intelligent applications. The toolkit allows users to acquire and run various language models, optimize and fine-tune models using both local and cloud compute, and efficiently deploy models to Azure AI Studio or other platforms using container images.
Microsoft Azure API Center, now generally available, provides a centralized solution for managing API sprawl, facilitating the discovery, consumption, and governance of APIs across their lifecycle. This unified inventory streamlines governance and accelerates API consumption. New capabilities in Azure API Management enhance the scalability and security of generative AI deployments, including policies for fair usage, optimized resource allocation, one-click import of Azure OpenAI Service endpoints as APIs, a Load Balancer for efficient traffic distribution, and a Circuit Breaker to protect backend services.
Azure Functions has also been updated to support the integration of AI, with features now in preview that include an extension for the Azure OpenAI Service. This extension enables customers to infuse AI into their applications, supporting new AI-led capabilities like retrieval-augmented generation, text completion, and chat assistants.
Lastly, GitHub is introducing GitHub Copilot extensions, developed in collaboration with Microsoft and third-party partners, in private preview. These extensions allow developers to customize their GitHub Copilot experience with preferred services such as Azure, Docker, Sentry, and more, enhancing the capabilities of GitHub Copilot Chat. GitHub Copilot for Azure showcases how building and deploying applications on Azure using natural language can significantly accelerate development velocity.
These announcements underscore the pervasive integration of AI within the Azure platform, enhancing the capabilities, performance, and security of applications across various domains.
Copilot advancements
As we all know, there’s no one Copilot, but a Copilot for each specific persona in turn built with several agents for different products or use cases. Oh well, we’re not there yet on a technical standpoint but that’s the direction. Ao let’s take a look.
Microsoft Copilot in Microsoft Azure for Azure architects and amins
Microsoft Copilot in Microsoft Azure, now available in preview to all customers, introduces new capabilities granting full autonomy to enable or disable Copilot within Azure tenants. This includes selectively granting access to specific user groups, thereby providing a secure, flexible environment that aligns with customers’ operational standards and encourages broader adoption. Key enhancements include app troubleshooting through conversational queries, allowing users to resolve issues by simply asking questions like, “Why is my app slow?” or “How do I fix this error?” Additionally, Copilot’s reach extends to Azure SQL Database, offering efficient management and operation of SQL-dependent apps by converting natural language inquiries into T-SQL commands.
Unified Copilot Extensibility with Copilot Extensions
Microsoft is unifying all Copilot extensibility concepts, including plugins and connectors, into a single construct known as Copilot extensions. These extensions will enhance Copilot by enabling new actions and customized knowledge integration. Developers can create Copilot extensions using Microsoft Copilot Studio or Microsoft Teams Toolkit for Visual Studio Code. This includes extensions from leading apps like Jira and Priority Matrix, as well as company-developed line-of-business extensions. IT admins will manage access to these extensions through the Microsoft 365 admin center, ensuring a tailored Copilot experience with the data, systems, and workflows used daily.
These extensions or copilots can integrate line-of-business data directly, automate long-running business tasks, leverage memory and contextual knowledge, and adapt based on user feedback. Copilot Studio also features connectors that integrate Microsoft Graph, Power Platform connectors, AI skills in Microsoft Fabric, and Microsoft Dataverse, streamlining the incorporation of organizational knowledge. Developers can publish Copilot extensions to Microsoft 365 and Microsoft Teams, customizing copilots with specific instructions and actions for enhanced user experiences.
Real-time Video Translation in Microsoft Edge
Microsoft Edge will soon feature real-time video translation, addressing accessibility barriers for users who are deaf, hard of hearing, or face language challenges. This AI-powered feature will translate videos across multiple platforms, such as YouTube and LinkedIn, into the user’s chosen language in real-time through dubbing or subtitles. Initially, translations will be available from Spanish to English and from English to German, Hindi, Italian, Russian, and Spanish, with plans to support additional languages and video platforms in the future.
Intelligent Recap Support for Meetings in Teams
Starting June 2024, Intelligent recap will be available for meetings with only transcription enabled, even without recording. This feature will provide AI-generated meeting notes, tasks, and name mentions, enhancing productivity and clarity in Teams Premium and Copilot for Microsoft 365. By leveraging AI, this tool ensures that key information and action items from meetings are captured accurately, helping teams stay organized and informed.
Quae Nocent Docent 