i think my ideal creative tool will be some kind of mix between a drawing / vector graphics program, an IDE, and an OS. ideally it’s extremely extensible, flexible, and lets you work with basic any kind of media and make it interactive.
it should be an infinite canvas. feels like 2d open world. i can put anything onto it. text, images, videos, static HTML, runnable code, full web apps. textures. lights. maybe even add physics to blocks and create custom layout systems. i should be able to drop in ML models on a whim and create new ones very easily. the entire canvas is searchable and powered by an LLM so it can help you perform bulk creative tasks and help you organize your work and automate tasks, like: “find a few pictures of sunsets and write a function to calculate the top 5 colors in all the pictures”. i should be able to design and program agents that can walk around the canvas and interact with the things i make and different websites like its an ecosystem to physically explore. i think its basically gotta be a huge programming environment where every piece of media is a different data type that can be fed into different programs and apps that all talk to each other.
let’s call a piece of media / content type a block. this is pretty common in most knowledge management tools. so like, a jpeg would be a block. an mp4 file. a code snippet. a selection / region of that snippet. an audio clip. an array of frequency values derived from the audio clip. a vector drawing of a logo. an interactive, embedded React app. all blocks!
in fact you can think of blocks as files, in a way, although a traditional text file could also be a block. the difference is that its easy to add arbitrary metadata to any block and change its content / manipulate its connected blocks based on that.
then, what we need is some sort of unix-ish I/O format that is shared between all blocks and used for communication and data manipulation. this will allow us to do things like connect text documents to images, UI controls to API calls, and actually hook things up.
interface Block {
id: string;
name: string;
description?: string;
createdAt: Date;
listeners: {
onMessage: (m: Message) => void
},
media: {
type: MediaType;
data: string | JSON | Blob;
},
userData: JSON;
}
interface Message {
from: id | tag;
to: id | tag:
data: JSON
}
once you have blocks and a basic messaging format, you need ways to interpret the messages. this is definitely inspired by smalltalk / hypercard / original OOP concepts. the key point here is being able to feed the messages into other applications directly, not just use it programatically. it’s almost like a social protocol for the objects stored in memory. we see this behavior in node-based graphics tools already but its isolated to the app. imagine if any app on your computer could talk to each other by default.
like, everytime i paste in a wikipedia link into my writing app, i could make it automatically scrape the data from that page. you might think, can you do that now? and the answer yes, but if and only if the apps have extensions you can build for them and publish. here, you could just do it without anybody’s persmissions or special tools. you just change things with some code or by piping data and messages in between existing apps.
i wanna be able to connect any code i write and its output to any other block. like, a python function that picks a random number, then uses that random number to find a random youtube video. or a function that picks a number from 1-10 and then uses that to pick the order of players in an online board game. these are simple but important use cases to show how we can be using more casual programming in our daily lives. and you should be able to run all bits of code. any language and runtime!
when i draw a circle, i should be able to give it physics. maybe i can even make the circle reactive based on the state or presence of other objects around it. maybe the canvas needs gravity so blocks can fall– but what if I just want a particular set of objects to have physics? i can add that circle to a rectangular frame, and put gravity inside that frame that makes the circle bounce like a ball. i can right click on the ball– and say “connect data to” and drag out its current x/y position to a bar graph. that rectangle will change in realtime relative to the ball. i can attach a “mass” metadata property to that circle, draw a rectangle underneath it, and program an onCollsion event for the rectangle that calculates the force being pressed onto it. this would be a great tool for teaching and playing with physics concepts!
i can write a story and grab data about its text, and the cursor(s) inside of it. if someone wants to create an screenplay editor here, they can– they simple drag together some text and page templates, create components out of them, and start writing. this is actually similar to the kinds of ad-hoc writing interfaces you can make in figma, but it’s not extremely robust for creating interfaces with dynamic numbers of text cells / inputs or recursive components (which are useful for creaitng things like good markdown rendering).
you can connect certain words to visual properties of the page it lives on. for example, i could make a text editor where every time you write the word “night” the screen dims to dark mode.
all standard transformation and layout functions from design programs are available: i can take pieces of code, text, etc and select them with a brush. i can trasnfrom them with a brush! i can paint on top of them, put visual filters on everything, make my code glow.
i can tie its visual state to the data tied in front of it. i can make a code editor block, create a trigger that says anytime the word “rainbow” is typed inside the editor, little rainbow emoji particles will fly out from my cursor on every keystroke.
i can create vectors and manipulate bitmaps with brushes, with code, whatever
i can feed my camera input as well and use that as a constant stream into filters, games, or other properties of the other blocks / objects on the canvas. for example, I can take my camera input, feed it into an ML image segmentation model, and connect its vertex data to circles on the canvas to create a live running 2d facial mesh. i can use my head tilt angle to navigate and pan the canvas, or to tie the size of my face in the camera to the z-index / zoom level of the canvas
the canvas essentially has its own runtime: its a massive 2D programming environment that emphasizes parallelism. it is big dynamic computation graph where its nodes are different media types and functions and UI that communicate between them. maybe it still has some structure and ultimately lies on a grid that can be used to easily index coordinates of objects, like a spreadsheet.
i can setup rulers and “tracks” for UI elements and blocks to be contained with in or create movement. so, i can make a slider.
i can create arbitrary views or interfaces that are dynamic, responsive, and inspectable. i can draw out a UI, like a slider, and connect it to code that manipulates it as well– creating bidirectional interfaces on the fly
i can create variations of any work, any sequence of actions, and create branches of event timelines. i can browse and edit the timeline and make clips of any point in history to be able to replay or recreate.
i can break this UI out of the screen and distribtue it across other computing mediums or devices. i can tie it to bluetooth. i can connect all sorts of hardware into input. the canvas itself has a streaming input that other data sources and devices can feed into.
its basically a completely self editing environment that comes prebuilt with all of the primitives you need to create any kind of custom interface or rebuild itself. in a sense its a turing complete tool, in a way its primitives and the rules they interact with is kinda like its DNA: it contains instructions on how the system can self replicate.
a lot of this is inspired by my work at replit, as well as figma, macos and unix, dynamicland, ink and switch, and more. i’ll probably add to this writing over time or make some more prototypes.
the pc hasn’t been invented yet!