How AI is Redefining User Interfaces

For both online and offline applications, user interfaces (UI) and application programming interfaces (API) have long been the standard modes of interaction. However, the burgeoning capabilities of AI in understanding natural language suggest a seismic shift in this paradigm. This brings into focus the potential of domain-specific languages (DSLs) as the new cornerstone for interfacing with AI-driven applications.

The evolution of interaction models is steering us towards an era where humans engage with their digital counterparts - digital doppelgangers - using natural language, both in text and voice. This development is gradually rendering traditional interaction methods like clicks, taps, and swipes obsolete. In several scenarios, the need for traditional UI elements like toolbars or even visual interfaces becomes superfluous.

This transition to a new interface paradigm is unfolding in stages. AI is currently making inroads into accessing systems via their APIs. By teaching a large language model (LLM) like ChatGPT about a system's API, it can interact with that API to gather or manipulate information. The next step involves LLMs controlling the visual user interfaces of applications. An LLM might be instructed to change a paragraph's colour in a word processor, either by using the application's API or by simulating mouse movements to select the text and change its formatting. While this approach adapts existing technology, future applications, especially those developed from 2024 onwards and built with AI integration as a fundamental component, will likely adopt a different strategy where DSLs play a pivotal role.

DSLs are specialised languages tailored to a specific domain. For instance, a DSL for a word processor like Microsoft Word would encompass elements such as headings, font colours, and text formatting. In this setup, what appears as red text on the screen might be represented in the DSL as [colour:red]hello world[colour]. An AI capable of interpreting and modifying this DSL could easily change text colours or perform other formatting tasks by simply altering the DSL commands.

This concept extends to more complex software like video editors. Traditional video editing involves manipulating multiple channels, layers, media types, and effects, typically stored in a file system. Converting this data into a DSL would allow an AI doppelganger to easily make changes. For example, if you want to adjust a video's aspect ratio, telling your AI doppelganger to change the canvas to 9:16 would result in it editing the DSL document to reflect this change. The AI's understanding of the DSL allows for more nuanced edits, like reformatting assets on the timeline to fit the new aspect ratio.

An example DSL for a video editor could be as follows:

// Define the Video Project
project "Summer Vacation Highlights" {
    resolution 1920x1080
    frame_rate 30fps
    output_format mp4
}

// Import Media
import "beach.mp4" as BeachScene
import "family_picnic.mp4" as FamilyPicnic
import "sunset.mp4" as Sunset
import "background_music.mp3" as Music

// Timeline Configuration
timeline {
    // Video Track
    track Video {
        clip BeachScene from 0:00 to 2:00 at 0:00
        transition fade to FamilyPicnic duration 1s
        clip FamilyPicnic from 0:00 to 1:30 at 2:01
        transition slide to Sunset duration 1s
        clip Sunset from 0:00 to 2:00 at 3:32
    }

    // Audio Track
    track Audio {
        clip Music from 0:00 to 4:32 at 0:00 volume 80%
    }
}

// Effects and Adjustments
effects {
    clip BeachScene {
        filter brightness 1.2
        filter contrast 1.1
    }
    clip Sunset {
        filter color_balance warm
    }
}

// Text and Titles
titles {
    title "Summer Vacation 2024" at 0:00 duration 5s style "Bold" size 24 color white
    subtitle "Fun at the beach" at 0:10 duration 4s style "Italic" size 18 color yellow
    credits "A family video produced by John Doe" at 4:00 duration 5s style "Regular" size 20 color white
}

// Output Settings
output {
    file "summer_vacation_highlights.mp4"
    location "/videos/family"
}

This approach closely resembles the principle of markdown. Markdown is a lightweight markup language designed for formatting plain text. It is easily readable and writable in its raw form and can be converted into structurally valid HTML.

In the same vein, DSLs could simplify complex software tasks. Just as markdown streamlines the creation of formatted text, DSLs could enable users to describe complex software actions in simple language, which the AI then interprets and applies to the DSL document. This method could revolutionise software interaction and development, making it more accessible and intuitive, much like markdown has simplified web content creation.