This is about
Design, Code
4 minutes
28 Feb 2025
This is about
Design, Code
4 minutes
28 Feb 2025
Exploring AI-powered art with simple text prompts
At 14islands, we’re always exploring new ways to push creative coding forward. Recently, we conducted an experiment using the Vercel AI SDK to integrate multiple Large Language Models (LLMs) for generating art purely from natural language prompts.
The Idea
Shaders are a powerful tool for real-time graphics rendering, but writing the required GLSL shader code can be daunting, especially for designers or developers unfamiliar with its syntax. We wanted to see if AI could bridge that gap by turning simple text descriptions into valid shader code.
The Experiment
We built a prototype where users could enter a natural language prompt (e.g., "a swirling galaxy with vibrant colors") and the system would generate a corresponding GLSL fragment shader. The AI-generated code was then rendered in real-time.
The following is a real embedded example of a prompt and the output it creates. You can expand to see the code or try to run your own prompt to see what comes back.
The prototype was built using popular opensource tools and the Cursor AI editor. Most of our time was spent designing a good user experience with help from our talented design team.
- Vercel AI SDK – A seamless way to work with multiple AI models.
- OpenAI, Claude, Mistral, Deepseek – Various LLMs to test their effectiveness at generating GLSL.
- WebGL – For real-time visualization of generated shaders.
- Next.js – To build an interactive web app.
Key Challenges
A few things stood out as we began generating shaders.
- Syntax Errors: Not all models will generate valid GLSL code all the time. But to be honest, we're quite impressed how stable the output is.
- Lack of artistic understanding: It seems like LLMs can piece together concepts well, but they don’t really ‘see’ the results of their code, making it unable to judge whether something is artistically pleasing.
- Performance Issues: Some generated shaders are computationally expensive, affecting real-time performance. The system prompt gives instructions to avoid deep loops, especially when doing raymarching, but sometimes the result is just too heavy.
It seems like LLMs can piece together concepts well, but they don’t really ‘see’ the results of their code, making it unable to judge whether something is artistically pleasing.
What we learned so far
- Claude seems to be the most consistent model. It's fast, which is good for iterating on ideas, and gives visually interesting results most of the time.
- It's interesting to follow the reasoning of Deepseek R1 where it explains how certain algoritms can be used. However, the quality of the output ususally doesn't match the theory behind it. It's also too slow for an iterative workflow.
- Mistral Codestral produced fast responses but with low creative expression - it also seem to be most likely to produce a broken shader.
- Fine-tuning the system prompt significantly improved output quality. (e.g., explicitly asking it to add interactivity and hinting which algoritms to use, such as Signed Distance Fields and Raymarching)
- Asking the LLM to create a plan first and then execute on the ideas seemed to improve the output and reduce compilation errors.
Takeaways
While AI is not yet perfect at writing shaders, it can be a powerful co-pilot for generative artists and developers. The experiment showed that LLMs can help accelerate shader prototyping, but manual tweaking is often required.
Running the same prompt several times and tweaking the LLM temperature setting can produce wildly different results.
What’s Next?
We’re excited to refine this approach further, perhaps by fine-tuning custom AI models on curated GLSL datasets. Another direction is interactive AI-assisted shader debugging, where the model can suggest fixes in real-time.
We’d love to hear from others experimenting in this space! Have you tried using AI to generate shaders? Let’s discuss!
NOTE: We've set a soft rate-limit on daily requests since we need to pay for API access to these models. Please let us know if you notice we've run out of the set budget and need to top up our balance 🙏
Droplets
Droplets
Sign-up to get the latest insight from 14islands. We send a newsletter only once every quarter with inspirational links and creative news. It's short and sweet. You can unsubscribe it at any time.
