Concept: Messages and Content

The http4k AI/LLM module uses sealed class hierarchies to provide type-safe, compile-time validated message structures with limited, well-defined options.

Message Types#

There are five main message types, each serving a specific purpose in the conversation flow:

sealed class Message {
    // Instructions and context for the AI
    data class System(val text: String) : Message()
    
    // Human input with multimodal content
    data class User(val contents: List<Content>) : Message() 
    
    // AI responses with optional tool requests
    data class Assistant(val contents: List<Content>, val toolRequests: List<ToolRequest>) : Message()

    // Results from tool executions
    data class ToolResult(val id: RequestId, val tool: ToolName, val text: String) : Message()

    // Extensible for provider-specific needs
    data class Custom(val attributes: Map<String, Any>) : Message()
}

Multimodal Content Support#

sealed class Content {
    data class Text(val text: String) : Content()
    data class Image(val image: Resource, val detail: DetailLevel) : Content()
    data class Audio(val resource: Resource) : Content()
    data class Video(val resource: Resource) : Content()
    data class PDF(val resource: Resource) : Content()
    data class Custom(val resource: Resource) : Content()
}

Messages can contain multiple content types - text, images, audio, video, PDFs, and custom formats. This enables rich, multimodal conversations with AI models.

Resource Handling#

Resources can be referenced by URI or embedded as base64 binary data, providing flexibility in how media is handled.

sealed class Resource {
    data class Ref(val uri: Uri, val mimeType: MimeType?) : Resource()
    data class Binary(val content: Base64Blob, val mimeType: MimeType?) : Resource()
}

Tool Integration#

Messages integrate seamlessly with tool calling - assistants can request tool executions, and tool results flow back as structured messages in the conversation.

scarf