By challenging traditional coding that typically relies on typing, Cornell University researchers created an interface that allows users to handwrite and draw in computer code. A pen-based interface called Notate allows users of computational digital notebooks to open drawing canvases and handwriting diagrams on lines of traditional digitized computer code.

Thanks to its deep learning model, the interface combines handwritten and textual programming contexts: representations in a handwritten diagram can refer to textual code and vice versa. For example, Notate recognizes cursive programming characters such as “n” and then associates them with their typed equivalents.

“A system like this would be great for data science, especially for drawing graphs and charts that then interact with textual code,” said Jan Aravjo, lead author of the paper and a PhD student in computer science.

“Our work shows that the current programming infrastructure is actually holding us back. People are ready for this kind of functionality, but API developers need to take this into account and support images and GUIs within the code.”

Aravjo also said that the work showed a new path by introducing pen-based artificial intelligence coding at a time when drawing tablets were becoming more widely used.

“Tools like Notate are important because they open new ways for us to think about what programming is and how different tools and presentation applications can change that perspective,” said Tapan Parikh, associate professor of computer science and co-author of the papers.