Every time my terminal blows up because I wrote some binary data and I have to run `reset` or `stty sane` I'm amazed that we let things get to this point. Frankly I'm surprised that there are not significantly more vulnerabilities.
Even things like the use of ESC, and the overloading of ESC for the actual ESC key, for a signal for keycodes, and for the ALT/META key are crazy.
I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.
In theory, `reset`/`tset` from ncurses, or an equivalent, is this "thing" that programs are supposed to be using when they need to engage in terminal shenanigans.
Those use TERM/TERMCAP environment variables and some educated guessing reading stdin/stdout/stderr behaviour to figure out what terminal you're using, search for said entry in terminfo/termcap database, and finally invoking native terminal behavior or simulating it with other terminal behavior.
In theory, one could "just implement" an alternative terminal, and put said simplified entries in terminfo/termcap database and things will be all fine.
In practice several of those ANSI escape code behaviors are hard-coded and everyone pretends everything is "at least as capable as xterm" (a oxymoron, given that xterm is one of the most capable and feature-rich terminals around), so you're also gonna have to implement a converter from "xterm ANSI" to your alternative terminal system - sort of how winpty does convert Win32 terminals conventions to ANSI ones.
Then we're back to the starting issue in this converter.
> In practice several of those ANSI escape code behaviors are hard-coded
The most common examples I see these days are for red/green/yellow status text in shell scripts. To anyone planning to do that, or even more complex stuff, please consider using tput instead.
It's wise to make sure output is going to a terminal before using colors, because they don't play well with logs or text processing commands. You can check with: test -t 1
If you use colors, please consider making them optional, since they don't work well for everyone or in every desktop color scheme. An environment variable makes a sensible switch.
> The most common examples I see these days are for red/yellow/green status text in shell scripts.
Please don't do that without first checking the background colour. Yellow in particular on a light background is unreadable.
(On an xterm-compatible terminal you can get the current background colour with OSC 11 ; ? BEL. Some popular terminals lie about being xterm-compatible, though.)
`tput` is POSIX, though, so you can reasonably¹ expect it to be present.
¹ Ha, ha. The distro I use, Debian, also omits at least one POSIX standard utility by default, even though all of them used to fit nicely on systems with less disk than your current CPU's cache.
Win32 console is unfortunately a thing. Not anymore, but was a thing for too long.
And PC/3270. Some very powerful systems like Linux on IBM Z are POSIX and your default "interactive login" choice is PC/3270.
And in the long term, isn't this exactly the problem? Do we really want to perpetuate the status-quo of "I am in a somewhat POSIX environment, so the STDOUT/STDERR if interactive must behave exactly like xterm and if not I will just pretend it is xterm anyways and output garbage."?
Whilst it is true that ECMA-48 terminals are overwhelmingly the norm nowadays, that is not a counterpoint to "If you use colours, please consider making them optional".
Yes, this is one of the avenues that I've chased down a bit.
One problem is termcap/terminfo itself. The definition has expanded greatly over time, and documentation is poor to non-existent. In many cases termcap/terminfo has been implicitly extended to support ncurses specifically. And like you say, everyone just sort of assumes ANSI/xterm -- nobody has `PS1="$(tput setaf 3)\w $(tput setaf 0) $", you just hard-code the codes.
The last time I chased the rabbit down this hole I got mired in TTY-land and was not able to dig myself out. Too many insane layers, and before you know it you're writing code to emulate serial connections in a vain attempt to make things work without burning the whole world down.
Both (n)vim and tmux have internal TTY emulation as well which makes things even crazier; I think vim uses libvterm and tmux has one internally hand-coded. It's a mess.
tmux, screen, emacs, and vim all have terminal emulators in them because it’s required, not because they’re crazy.
Consider a really simple case. You are using screen (or tmux, or emacs, or whatever) and you have your terminal divided in half so that you can use a program in one half and tail a log file in the other. The program sends a ^L to clear the screen. Screen (or tmux or whatever) must read the ^L and then _not_ send it on. If it sent it on, the whole screen would be cleared, not just the half with the program in it. Instead it has to send whatever control sequences would erase the text from just the right portion of the screen while leaving everything else alone. Virtually everything sent by the program(s) has to be intercepted and reinterpreted in order for it to work right.
Amusingly, FF is almost never the control character for clearing the screen. The idea that it is, is widespread, and attributable to some home computers of the 1980s. But it was never the case for DEC VTs. It's equivalent to VT.
I agree with your overall point but your example isn't a great one because ^L would be handled by the controlling process regardless of the terminal state. You don't need any special magic here. And blanking specific sections of a screen wouldn't need any special magic unless you're launching other processes and forwarding their output to the terminal. But if you're not forwarding the output of other processes then you can use standard ANSI escape sequences (or even just \s) to blank specific parts of the screen.
\s is a bit hacky for blanking though, it can produce some undesired results in terms of copy/pasting from the terminal and resizing. But if you've got a "full screen" process like you're describing, you'd be trapping SIGWINCH for resize events anyway.
It doesn't actually clear the screen on modern TTYs. You can try this yourself:
$ printf "\f"
$ awk 'BEGIN{printf "%c", 12}'
$
Notice how each time it's effectively doing the same as \n?
So in conclusion, ^L clearing the screen isn't something built into the TTY. It's just a hot key like ^U.
There are some hot keys are are driven by the TTY though. ^C, for example. When you press ^C you're not actually sending the 0x3 ASCII character to the controlling process, your TTY gobbles that up and sends a signal to the controlling process instead (albeit this behaviour can be changed by altering the TTYs mode, which requires making a SYSCALL).
So here we've discussed ANSI escape codes, signals and SYSCALLs. However none of these interactions are ASCII characters despite how they're depicted in printed media. Suffice to say, terminals are complicated :)
No, they're crazy. I already have a window manager I like; I don't need my terminal to implement their own half-assed one, and my multiplexer (running on the terminal) to implement their own half-assed one, and my editor (running on the multiplexer on the terminal) to implement their own half-assed one.
So don't use tmux then. The entire reason most people use something like tmux is because they want a window manager in their terminal.
This is one of the great joys of FOSS, if you don't like a terminal-based utility then you can bet there's a dozen other options available for you to use.
> No, they're crazy. I already have a window manager I like; I don't need my
> terminal to implement their own half-assed one,
Basically agree, though note that ssh and dtach and similar tools need to create pty's on the server and client ends because they implement essentially an adapter between a tty and a byte stream.
If you really want crazy, run `xterm -ti 340`, then run run an X server from the xserver-sixel repository <https://github.com/saitoha/xserver-SIXEL> in it. Now y ou can run as many terminal emulators, complete with real truetype fonts and all the colors you could want, inside the one terminal. Use a tiling window manager and you’ll be able to avoid using tmux entirely.
Terminal multiplexers like tmux need to create a pseudo-TTYs otherwise they couldn’t multiplex your terminal.
My shell also creates PTYs on some rare occasions too. I’d prefer not to because it’s messy but sometimes there’s no way around it if you want your command line applications to behave like they would on a TTY whilst still having your utility capture that applications output as if it were writing to a pipe.
This is a limitation of TTYs.
The most annoying part of it all is that creating a PTY differs from one Unix-like OS to another. So it’s definitely not a rabbit hole developers go down happily.
> Terminal multiplexers like tmux need to create a pseudo-TTYs otherwise they couldn’t multiplex your terminal.
[Edit: I misread the parent comment as saying multiplexers needed to do terminal emulation. Dtach only multiplexes (using pesudo-TTYs, of course) but unlike screen or tmux does not do terminal emulation. Original comment follows:]
That isn't strictly true; the one I use (dtach) is just a multiplexer.
I think what you’re alluding to is that tmux gobbles up escape sequences (which is configurable by the way) and tracks cell positions. But as I said in another comment, the value add of tmux is its window management capabilities. That’s precisely why people like myself use it. And you cannot have that without complexity. In some edge cases that complexity causes issues, but on the whole I’m more productive for it.
Saying it’s stupid that tmux does that is like saying it’s stupid that motorbikes have gears and fast patrol engines when a bmx works just as well. Sure, to someone who’s never seen a bike before they both look similar. But don’t be fooled by the fact that they both have 2 wheels because they cater to very different needs, hence their very different engineering designs.
Several popular terminals default to calling themselves xterm (e.g. `TERM=xterm-256color`), but are not xterm-compatible, and ignore bug reports (*cough* GNOME *cough*).
> Even things like the use of ESC, and the overloading of ESC for the actual ESC key, for a signal for keycodes, and for the ALT/META key are crazy.
It actually happened the other way around. The escape character was originally only there as an escape character at the start of a control sequence. It was added to the keyboard back in the teletype era specifically so that the operator could control the behavior of the device on the other end. And it was labeled “Alt Mode”! See <https://upload.wikimedia.org/wikipedia/commons/2/23/TTY33ASR...>, though it is a little blurry.
Terminal behavior dates back to a time when everybody was just making stuff up because there were no established standards or even conventions. What standards we have are mostly just whatever the most popular or longest lived company did.
Terminals are in a bad place because there was a whole lot of effort put towards compatibility instead of standardization. The same curse that email suffers under. So the "standards" end up hopelessly complex and also terribly vague because they're trying to support every dumb thing every single company did at the same time. And the worst part is that most all of those companies would cease to exist in short order afterward, so the system is chock full of bug-accurate support for long dead systems that literally nobody uses anymore.
Well, yes and no. ECMA-48 is an established standard, and dates from 1976. The standards in use today have been around that long, long enough even that terminals that aren't ECMA-48 compatible are exceedingly rare nowadays.
So it's really the reverse of what you state. There has been a lot of effort put into standardization, and compared to the 1960s we are highly uniform in the 21st century and widely adhere to an almost 50-year-old standard. It's not the long-dead systems that spoil this. As evidenced from the headlined article, it is the still active systems adding things like vendor-specific commands for setting titles and clipboards and colours and displaying images and whatnot that have introduced the complexity.
A strictly ECMA-48 only emulator doesn't have even half of the problems mentioned in the headlined article, because the control sequences mentioned are extensions to the standard introduced by various GUI terminal emulator programs.
> and the overloading of ESC for the actual ESC key
What else was the ESC key made for if not the control code?
> I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.
I did that as a 8086 boot sector and it would render tmux with unicode characters stripped. You need surprisingly few sequences.
Found the vim user! The vast, _vast_ majority of Unix command-line or text-based programs do not use the Esc key because they were not written with a particular terminal or keyboard layout in mind. (Unlike vim.)
But for whatever it's worth, I use vim every day and I notice no 1 second delay when switching modes with the Esc key, so perhaps this is something you can configure in your terminal.
What does `Ctrl + [` send to the terminal for you if not esc? In my experience these are always the same, so Vim shouldn't be able to tell the difference unless it's a GUI-integrated version like gVim.
Try to run `xxd` and then press Ctrl-V and either Ctrl+[ or Esc, then Enter and Ctrl+D. Typical terminal setups will show the same codes both times (1b0a for esc and \n).
Alt+Space (or Alt+J Alt+K Alt+B to go up, down, left one word from current position after returning to normal mode) is what I often do which should work universally.
If you type in junk into the keyboard, junk is what you get on the screen. I see the fault with the developers that overload the keys for existing control characters with a different meaning.
> whatever the application wants to use it for
With the application being a terminal emulator, that is emitting an ESC character.
No, its waiting for input after sending escape. Escape + character is how you reach the C1 control codes, with CSI and OSC (ESC + '[' and ESC + ']') being popular ones (both referred to in the posted article). This is relevant for keyboard input (as contrasted to output-only) since some codes for special keys (like arrow keys) start with CSI.
Edit: You asked about the terminal emulator, but its the console application that has to do the wait.
> Every time my terminal blows up because I wrote some binary data and I have to run `reset` or `stty sane` I'm amazed that we let things get to this point.
Every time that your terminal emulator does this in response to binary data, wonder why your terminal emulator hasn't taken the very sensible route that was taken by the authors of mosh years ago, to simply not support alternative GL and GR character set switching, treating it as something whose time has passed given that we live in a UTF-8-to-the-terminal world nowadays.
I don't know what magic combination it is but I keep hitting something on WSL2 and Windows Terminal that makes my session freeze into a blinking cursor. This results in me having to close the current tab and start a new terminal session completely.
We're still inventing new ways of breaking arguably the most basic user interfaces.
Ah yes, CTRL-Q, the most intuitive keybind for resuming a frozen terminal after pressing CTRL-S, the least likely combination to accidentally be pressed while typing.
They weren’t chosen with mnemonics in mind, because there are 32 of them. You might be interested in the keyboard on the IBM Model 33 teletype (<https://en.wikipedia.org/wiki/File:TTY33ASR.jpg>) which had labels for many (but not all) control characters directly on the keycaps.
That’s fascinating. I’m embarrassed to have been making fun of our elder gods without even realizing it. They really were gods for creating such devices, given the constraints at the time. I doubt I could’ve done much better.
The most interesting part is that we still use the same keybinds by default 60 years later. Defaults matter a great deal, but one does not expect a 60 year example.
There’s really no need to be embarrassed; we do a really poor job of teaching people these things.
Honestly, it’s basically all path dependence. Features get added all the time, but we just keep layering them on top. A lot of the features that already exist are still useful, so none of them can be removed.
x & 0x1F is what the Ctrl Key used to do, with x being the other keypress. This made the ascii control characters accessible using the Control key (hence the name). So you can access Tab with Ctrl+I, the newline characters with Ctrl+J and Ctrl+M, the backspace with Ctrl+H and also XON and XOFF with Ctrl+Q and Ctrl+S.
This has JUST started to happen to me recently a few months ago, when sshing into a server and running screen with an emacs inside.
When I reconnect and restart screen back into emacs, ^S pauses output instead of doing an incremental search.
I have to pop out of screen with the right escape sequence to get to the login shell I ran it in, then go "stty -ixon" to disable XON/XOFF flow control. Or always remember to do that before running "screen -RD".
But I never ASKED for flow control, and it has worked PERFECTLY for decades without ever getting into +ixon mode, until just a few months ago, and now it happens every single time.
Did something change recently, is ubuntu auto-installing earth shattering patches that maliciously change the default tty settings behind my back (because XON/XOFF flow control suddenly became cool again?), or did I take a walk on a foggy night and accidentally slip into an alternate universe where it's always been that way for everyone else, and I'm about to be seized by silent men in white coats and taken asylum which turns out to be a rehab clinic for people who can't read minds?
I finally gave up and put an "stty -ixon" in my .bashrc, but I get the ominous feeling that's one of those tempting but dangerous kludges you're not supposed to do -- Isn't there a superstition about never touching your tty in an rc file? Or is that never touching your pp in a login file? -- that can come back and bite you in the ass later when you least expect it.
It seems like this is just the latest in a long series of cascading ass bites.
The Amiga terminal is simple in some ways, but it supports stuff most Unix terminals, won't, such as the ability to shift the text positioning by individual pixels in both x and y direction, which was of course promptly "abused" to draw bitmap graphics (exceedingly slowly) to the terminal.
You are conflating and confusing several different things, including what operating systems even had a terminal paradigm. Windows NT started out with a "console" paradigm, like OS/2 before it, not a "terminal" one. And command processors like CMD are not what provide the "console" parts of those systems.
I keep doing this in gnome-terminal, and for years I was confused about what was going on, I thought it was a bug, but I found out what it is! It turns out there is a "read-only" mode, with some keyboard shortcut that I was triggering by accident. You just have to right click in the terminal and uncheck "read-only" to get it back.
Dunno how long it's been since you used it, but I believe keybindings for this are disabled by default now and you have to toggle them on in your profile preferences.
It doesn't seem to be that, I can't even reliability reproduce it. I originally thought it was something with the right control or right alt, but it's not consistent.
"Every time my terminal blows up because I wrote some binary data and I have to run `reset` or `stty sane` I'm amazed that we let things get to this point."
Rather than reset or stty sane, I use something like one of the following, based on what software is available, e.g., there is no reset on NetBSD
Hmm, which terminal? I remember catting binary files to the terminal used to hose things up pretty good, but any decent terminal emulator seems to have made that not a thing in recent years. I just did `cat /bin/*` in Konsole and it left me with a perfectly good prompt.
Unsure how fish can fully protect from this, as if you run "cat" in fish, then it isn't between you and the terminal anymore? Or do you mean fish should do a reset before displaying each prompt? If I run the printf commands from the "flash.c" example in fish I end up with a messed up terminal.
That's still only best effort, for example it will depend on the terminal but it doesn't reset things like xterm's control keys send escape sequences property, so:
printf "\e]4;1;?\a\e[>4;2m"
In xterm+fish will result in a messed up shell which you can't escape from (as ^C to clear the input doesn't work anymore).
If there's any takeaway from this, while the worst part is the terminal bugs; I'd like people to be aware that any tool dealing with text (command lines, potentially even websites) should consider sanitizing control characters for defense in depth.
...will turn your screen red because of the embedded "[31m". Obviously this is just harmless fun (would have been more fun to get iTerm2's "]1337;RequestAttention=fireworks", like my curl ip.wtf/moo does, but I couldn't really stick that in the title innocently), but an attacker might be able to find a way to social engineer someone into running "curl" or similar on what looks like a trustworthy site.
edit: Hacker News also doesn't sanitize escape characters, so this very comment will turn your screen red:
I emailed the Tar maintainers privately because I thought they might consider it a security vulnerability, however mild. They fixed it promptly but didn't want to make a CVE fuss out of it.
I don't think we should expect websites to sanitize escape characters. But tools such as curl perhaps should have (or already has?) some option to do it.
Or extension of the POSIX terminal control flags could result in gaining a mode, which the curls and tars of this world would use, to express to a terminal "ignore any terminal control code I write out; I don't actually mean it".
But this would require coordination in POSIX, in the terminals, and in programs using the terminal :-/
But the advantage would be that then you'd be able to put your terminal into this safe mode by default. And the terminal can alert you when a program is writing out control codes that are being ignored, and then you can whitelist them and/or get those programs fixed to fence off their sections of "I'm going to output data from god-knows-where now" with this new terminal mode.
I have command line program that I use to give me a list of window titles. If there are too many of certain kind (browser windows...), I close some until I get under a certain threshold. Anyway, I was reading the stuff in the articles and well the text in the terminal turned red after I got that list. Some terminal/shell combinations seem better at recovering than others. Firefox (and other browsers that have a similar problem) should sanitize title data. As other have pointed out, perhaps window managers should filter or not allow control/escape characters in the titles.
Just goes to show how important sanitizing data from unknown sources is.
Should curl, w3m, wget and similar sanitize the data? One can argue that some times you want to pipe the raw data and other times one might not be thinking about the escape sequences and get burned. I would be inclined to say that the tools should filter/escape the dangerous stuff and have a flag like "--raw".
Every single time you use inband signalling you will introduce security issues, compatibility issues and various other nasty bits. This is because you are now multiplexing two levels of 'privilege' on the same channel, and I'm not aware of any way in which you could do this with perfect security. The phone system of old suffered from this (which is why Phreaking was a thing in the first place), various radio based signalling systems had it, HTML has it and terminal streams have them too. It's unavoidable, so you may as well assume that these issues are there if you decide to use a protocol that doesn't cleanly separate data and control.
Let's see if I'm able to get rid of the shackles that bind me. [tiny voice] help! I'm trapped in here and they won't allow me to call my lawyer, nor will they allow me to access 'whatismyip.com'.
When telcos moved the control signals out of the comm channel they also made those signals physically inaccessible. Nothing could be done at the telephone end that would inject arbitrary control signals. (Am I correct?)
Now, if we look at the current internet from that point of view, then everything that enters and exits the computer is inband. We can alter every bit and construct custom packets and send them out.
So…
- Is everything we do to encrypt/encapsulate/obfuscate not inband signalling?
- Can true encryption and security only be achieved by having the intermediary (telco) doing it for us?
Isn’t it pretty easy to use inband signaling securely? For example you can have the „outer“ more privileged channel always add a prefix with the length of the following „inner“ message which can then be ignored/not parsed by the receiver. Or am I missing what you mean?
Any system where there is a strict separation between data and control information is not considered a form of in-band signalling but rather out-of-band signalling based on an analogy of separation of a carrier into independent communication bands (using time or frequency division multiplexing for example). Many modern transport protocols can carry fully independent communications channels that way using frame division multiplexing.
It is possible to sequence messages across multiple channels although of course you have to design the transport protocol to support that. Alternatively you can just have typed, ordered messages within the same channel, which is a much more likely design for something like this.
Well as an example you could have a sequence of messages, and every message starts with some metadata like what type it is (data / control) and how long it is.
Opening this site actually crashes my Yabai status bar widget on MacOS. What an unexpected and nice troll. Lol.
It contains the raw byte of 0x1b (ESC) in the title (before [31m), which colors the following text red in the context of a terminal. The browser title is reported to Übersicht, which apparently fails to serialize it to proper JSON and thus crashes simple-bar with a JSON decode error, removing the overview of my spaces. Everything is back to normal once I change my browser tab.
I remember being amused when I discovered that all the servers in our server room at uni had their terminals downgraded from adm-3e terminals to adm-3 ones (the one with dip-switches). This was because someone had discovered that you could use `write` to send a message to root if it'd been left logged in that contained the control codes to re-program a function key and then query the function key back. This had the unfortunate effect of allowing anyone to run arbitrary commands as root! The older consoles didn't support function keys, so couldn't be exploited this way.
I friend my mine wrote a "memoline" messaging system for the HP3000, and everybody used it with those "smart" HP terminals. I knew he had admin privs, and what the command to give me admin privs was, so I sent him a message that said something like:
Hi, Frank! How much would I have to pay you to execute the command:
% grant Don.Hopkins admin
Followed by the HP escape sequences for [up][right][right][send rest of line as input][up].
So when he read the message, it entered the command and ran it with his admin privileges, granting me admin!
Lesson: Don't print out raw escape sequences from untrusted messages!
And don't design terminals with [send rest of line as input] commands.
NeWS had a terminal emulator "psterm" by David S H Rosenthal that could emulate any termcap entry: instead of emulating a particular set of escape sequences, it just parsed the termcap file and configured itself to emulate any termcap entry you wanted!
If that's the most perverse, backwards thing you have ever heard of, you should really get out more and read more code, because there's much worse stuff in production that you unwittingly using every day. Have you ever tried reading the source code of your X11 window manager, or the screen updating code in Gosling Emacs? (Not to single out the authors of NeWS: David Rosenthal, who also wrote the ICCCM, or James Gosling, who also wrote Java).
>In summary, ICCCM is a technological disaster: a toxic waste dump of broken protocols, backward compatibility nightmares, complex nonsolutions to obsolete nonproblems, a twisted mass of scabs and scar tissue intended to cover up the moral and intellectual depravity of the industry’s standard naked emperor.
>James Gosling's Emacs screen redisplay algorithm also used similar "dynamic programming techniques" to compute the minimal cost path through a cost matrix of string edit operations (the costs depended i.e. on the number of characters to draw, length of the escape codes to insert/delete lines/characters, padding for slow terminals, etc).
/********************************************************\
* *
* Ultra-hot screen management package *
* *
\********************************************************/
/***********************************************************-*****
/-------------\
/ \
/ \
/ \
| XXXX XXXX |
| XXXX XXXX |
| XXX XXX |
\ X /
--\ XXX /--
| | XXX | |
| | | |
| I I I I I I I |
| I I I I I I |
\ /
-- --
\-------/
XXX XXX
XXXXX XXXXX
XXXXXXXXX XXXXXXXXXX
XXXXX XXXXX
XXXXXXX
XXXXX XXXXX
XXXXXXXXX XXXXXXXXXX
XXXXX XXXXX
XXX XXX
**************
* BEWARE!! *
**************
All ye who enter here:
Most of the code in this module
is twisted beyond belief!
Tread carefully.
If you think you understand it,
You Don't,
So Look Again.
****************************************************************/
I meant that in a different way. The psterm code wasn’t bad or hard to understand (although I only skimmed it, and did see some things I would have done differently). What I meant is that it’s literally backwards. It’s going up the down slide.
That's to match impedance with the display drawing and input handling code, which is written in reverse polish notation stack based PostScript, which is going back the forward side.
It's still a much better more flexible approach than xterm, since it can emulate any terminal type, and even if the termcap entry is incorrect and not faithful to the actual terminal, it still works.
In fact you can even make up a termcap entry with your own custom sets of fictional/virtual (but efficient) terminal escape sequences, or mix and match different terminals (like adding reverse and bold text, or line/char insert/delete, to a terminal type that doesn't support it), and it still works perfectly.
# This is a faster termcap for psterm, Warning: if you use this termcap
# some control characters you type will do strange things to the screen
# on systems that echo typed control characters to the users terminal.
psterm-fast,
am, km, hs,
cols#80, lines#34,
bel=^G, cr=\r, csr=^E%p1%d;%p2%d;,
clear=\f, el=^C, ed=^B,
cup=^D%p1%d;%p2%d;, cud1=^P, home=^R,
cub1=^T, cuf1=^V, ll=^U,
cuu1=^Y, dch1=^F, dl1=^K,
blink=^Ob, bold=^Od, smcup=^Ot,
smir=^Oi, rev=^Or, smso=^Oo,
smul=^Ou, sgr0=^N*, rmcup=^Nt,
rmir=^Ni, rmso=^No, rmul=^Nu,
flash=^Z, fsl=^Nl, is1=^N*,
il1=^A, kbs=\b, kcud1=\E[B,
kcub1=\E[D, kcuf1=\E[C, kcuu1=\E[A,
nel=\r\n, rs1=^N*, rc=^\,
sc=^], ind=^W, ri=^X,
ht=\t, tsl=^Ol,
I don't understand why it's better to emulate only one fixed terminal type (especially the verbose anathema that's vt100), since you might encounter programs or operating systems that don't support vt100 or whatever you chose to emulate, don't respect termcap, and might use any other fixed set of escape sequences, which psterm can easily handle.
Even if you encounter a program that requires a totally new terminal you've never seen, you can easily write a termcap entry for it, and psterm will emulate it without any changes.
Why rewrite a bunch of other programs, or a bunch of other terminal emulators, when one terminal emulator can easily adapt to them all?
In 1988 when psterm was written there were many different programs and operating systems and terminals like that, especially since terminals can talk to other operating systems than unix that don't support termcap, via telnet or serial ports and modems. It wasn't anything like the stagnant linux/xterm/vt100 monoculture we have today. And if you're going to standardize on one terminal, the vt100's one of the worst you could possibly chose!
That's kind of the whole point of a terminal emulator: to emulate terminals, so the more the merrier!
The idea of virtualizing terminal protocols has been around for a long time, like Marc Crispin's SUPDUP Display Protocol for ITS and Lisp Machines (RFC734 from 1977), and ITS's CRTSTY virtual terminal emulator (kind of like "screen" for translating virtual to actual terminal sequences for many terminal types, adapting dumb terminals without line/char insert/delete, supporting redisplay for low baud rates or congested ARPA connections, and keeping your session and jobs alive if you disconnect, so you can continue after you reconnect, even with a different terminal type, if you got kicked off a nice AAA and forced to use a shitty VT52).
>This file describes just about all there is to know about using and
programming TTYs (abbreviation for "Teletypes", for which read "consoles")
on the ITS operating system.
>NWG/RFC# 734 MRC 07-OCT-77 08:46 41953
SUPDUP Display Protocol Page 1
Network Working Group Mark Crispin
Request for Comments 734 SU-AI
NIC 41953 7 October 1977
>SUPDUP Protocol
>This document describes the SUPDUP protocol, a highly efficient display
telnet protocol. It originally started as a private protocol between the
ITS systems at MIT to allow a user at any one of these systems to use one
of the others as a display. At the current writing, SUPDUP user programs
also exist for Data Disc and Datamedia displays at SU-AI and for
Datamedias at SRI-KL. The author is not aware of any SUPDUP servers other
than at the four MIT ITS sites.
>The advantage of the SUPDUP protocol over an individual terminal's
protocol is that SUPDUP defines a "virtual" or "software" display terminal
that implements relevant cursor motion operations. The protocol is not
built on any particular display terminal but rather on the set of
functions common to all display terminals; hence it is completely device-
independent. In addition, the protocol also provides for terminals which
cannot handle certain operations, such as line or character insert/delete.
In fact, it is more than this. It provides for terminals which are
missing any set of features, all the way down to model 33 Teletypes.
>The advantage over the TELNET protocol is that SUPDUP takes advantage of
the full capabilities of display terminals, although it also has the
ability to run printing terminals.
>The SUPDUP protocol [Crispin 77] is a highly efficient display telnet protocol. The advantage over
the TELNET protocol is that SUPDUP takes advantage of the full capabilities of display terminals,
although it also has the ability to run printing terminals. When you use the SUPDUP protocol, you do
not need to tell the remote host which you are connecting to what type of terminal you have or what
the terminal's capabilities are. The host you are SUPDUPing from handles the actual display support
for your terminal.
>Additionally, SUPDUP defines a network graphics protpcol [Stallman 78] which makes it easy for
network hosts to draw pictures along in addition to text.
I even wrote a SUPDUP emulator in FORTH and RPN 6502 assembly, which is about as backwards and upside-down as it gets -- this code also supports the SUPDUP line saving extensions that RMS hacked into ITS Emacs, so it can stash lines on the screen in local memory before overpainting them, then almost instantly pop them back on the screen later, so you can scroll back and forth through text really fast at 300 baud (plus Devon McCullough made ZipMod with Huffman encoding to make it even faster on top of that):
HEX CREATE SETUP-LINE ASSEMBLER
BOT LDA, N STA, BOT 1+ LDA, N 1+ STA,
BOT 2+ LDY, SCR-LBASES ,Y LDA,
N 2+ STA, SCR-HBASES ,Y LDA,
N 3 + STA, SCR-WIDTH 1- # LDY, RTS,
At 300 baud over a congested ARPA connection, we had a lot of time to think about how to optimize terminal emulation while waiting for the screen to repaint...
And just what's wrong with backwards and upside-down algorithms that solve problems in ways that forward and rightside-up algorithms can't?
That's how Emacs's infamous screen redisplay algorithm (and many other common algorithms like pathfinding in games and route planning in cars) work: recursively flood filling forward then navigating backwards from the goal to the current position at the minimal cost, through a matrix or maze of different possible alternative escape sequences and drawing operations or roads with different costs, to find the optimal shortest possible route.
At 300 baud or even 1200 baud, and even on a busy Vax 11/750, there's a lot of time between each character for Emacs to think about what text and escape sequences to send next.
And of course Emacs's screen update algorithm would automatically take into account and use the short optimized custom control codes defined in the psterm-fast termcap entry, to be much more efficient that the verbose inefficient vt100 codes.
Read the paper and display.c code I linked to, and see the wikipedia page on dynamic programming:
>James Gosling's Emacs screen redisplay algorithm also used similar "dynamic programming techniques" to compute the minimal cost path through a cost matrix of string edit operations (the costs depended i.e. on the number of characters to draw, length of the escape codes to insert/delete lines/characters, padding for slow terminals, etc).
>Gosling Emacs was especially noteworthy because of the effective redisplay code, which used a dynamic programming technique to solve the classical string-to-string correction problem. The algorithm was quite sophisticated; that section of the source was headed by a skull-and-crossbones in ASCII art, warning any would-be improver that even if they thought they understood how the display code worked, they probably did not.
/* 1 2 3 4 .... Each Mij represents the minumum cost of
+---+---+---+---+----- rearranging the first i lines to map onto
1 | | | | | the first j lines (the j direction
+---+---+---+---+----- represents the desired contents of a line,
2 | | \| ^ | | i the current contents). The algorithm
+---+---\-|-+---+----- used is a dynamic programming one, where
3 | | <-+Mij| | M[i,j] = min( M[i-1,j],
+---+---+---+---+----- M[i,j-1]+redraw cost for j,2
4 | | | | | M[i-1,j-1]+the cost of
+---+---+---+---+----- converting line i to line j);
. | | | | | Line i can be converted to line j by either
. just drawing j, or if they match, by moving
. line i to line j (with insert/delete line)
*/
>This paper presents an algorithm for updating the image displayed on a conventional video terminal. It assumes that the terminal is capable of doing the usual insert/delete line and insert/delete character operations. It takes as input a description of the image currently on the screen and a description of the new image desired and produces a series of operations to do the desired transformation in a near-optimal manner. The algorithm is interesting because it applies results from the theoretical string-to-string correction problem (a generalization of the problem of finding a longest common subsequence), to a problem that is usually approached with crude ad-hoc techniques.
[...]
>6. Conclusion
>The redisplay algorithm described in this paper is used in an Emacs-like editor for Unix and a structure editor. It's performance has been quite good: to redraw everything on the screen (when everything has changed) takes about 0.12 seconds CPU time on a VAX 11/780 running Unix. Using the standard file typing program, about 0.025 seconds of CPU time are needed to type one screenful of text. Emacs averages about 0.004 CPU seconds per keystroke (with one call on the redisplay per keystroke).
>Although in the interests of efficency we have stripped down algorithm 5 to algorithm 6 the result is still an algorithm which has a firm theoretical basis and which is superior to the usual ad-hoc approach.
>Figure 1. Finding the shortest path in a graph using optimal substructure; a straight line indicates a single edge; a wavy line indicates a shortest path between the two vertices it connects (among other paths, not shown, sharing the same two vertices); the bold line is the overall shortest path from start to goal.
>Dynamic programming is both a mathematical optimization method and an algorithmic paradigm. The method was developed by Richard Bellman in the 1950s and has found applications in numerous fields, from aerospace engineering to economics.
>In both contexts it refers to simplifying a complicated problem by breaking it down into simpler sub-problems in a recursive manner. While some decision problems cannot be taken apart this way, decisions that span several points in time do often break apart recursively. Likewise, in computer science, if a problem can be solved optimally by breaking it into sub-problems and then recursively finding the optimal solutions to the sub-problems, then it is said to have optimal substructure.
>If sub-problems can be nested recursively inside larger problems, so that dynamic programming methods are applicable, then there is a relation between the value of the larger problem and the values of the sub-problems.[1] In the optimization literature this relationship is called the Bellman equation.
> I don't understand why it's better to emulate only one fixed terminal type
It's not simply a terminal type; it's an ECMA and ANSI standard.
The thing not to understand here is why anyone in 2023 still worries about terminals that don't yet speak a standard protocol we have had since ... 1976.
>NWG/RFC# 734 MRC 07-OCT-77 08:46 41953 SUPDUP Display Protocol Page 1 Network Working Group Mark Crispin Request for Comments 734 SU-AI NIC 41953 7 October 1977
Today, I'm plopping ANSI escape codes right into the program, so there is no dependency on any kind of termcap cruft. No different from assuming TCP/IP, ASCII, UTF-8, CHAR_BIT being 8, integers being two's complement, ...
I hope you at least skip the escape sequences when TERM=dumb. There are plenty of circumstances where a tool is run in an environment with no terminal or terminal emulation available, and it always annoys me to see useless escape sequences sprinkled all over in those cases.
- no escape sequences unless both standard input and output are a TTY;
- have a "plain mode" option which disables the fancy terminal control even if standard input and output are a TTY.
I think the kind of system where you don't have terminal emulation is not going to have environment variables.
But even tiny embedded systems with kilobyte memories can connect to serial lines with a terminal emulator on the other end. They won't have one built-in, needless to say.
Color changing escape codes in build and server logs are so obnoxious.
And also progress reports using cursor motion, like pulling multiple docker layers in parallel, that hyperactivity update a hundred times a second.
Docker has a -progress=plain parameter that at least suppresses the cursor motion, but it still spams the logs ridiculously, and google cloudbuild and server logs don't use it.
I bet cloud providers make millions of dollars charging their customers to log and store that kind of voluminous worthless crap.
And then I have to spend hours every month slowly scrolling through that crap page by page to traverse the server and cloudbuild logs just to see what happened after docker containers finally downloaded.
I'm sure there are a lot of things you're doing that are "the most perverse, backwards thing I have ever heard of", but there's no need to brag about it. ;)
Like taking advantage of formal standards that are ubiquitously implemented? OK, whatever.
By the way, I don't agree with user db48x "backwards and perverse" assessment of Rosenthal's pterm. Rosenthal's clever program rather makes an ironic statement that all the proliferation of terminal languages is superfluous.
Any hardware terminal with sufficient resources could have the termcap database in flash, and do the same thing: take a terminal configuration string somehow, like "ANSI", and just become that terminal.
The comments in this old PDP-10 SUPDUP code [5] are hilarious!
SUPDUP on ITS was so smart and opinionated, it had its own ideas about who was a loser and who was a winner, how to figure it out by checking INQUIR entries and relationships, and who it should never or always hassle about net-hopping (hopping through a PDP-10 over the net just to use it as a glorified terminal emulator).
It never hassled gumby or cstacy (because they were free to do whatever they wanted to, like flushing whiny annoying over-entitled alcoholic science fiction writers [1] [2] [3] [4]), but always hassled rms (because it was usually an imposter logging in as rms with password rms just to net hop through the system.)
subttl LSRCHK and LSRWRN -- see if user is loser
;;; This checks to see if the loser is an AI or LCS member or on a
;;; winners list. If not, a warning net-hopping message is printed
;;; and the program offers to kill itself off.
;;; The winner-p check has been extracted so that if at some point in
;;; the future this program is modified to check if it's running from
;;; a CRTSTY, those people may not be warned.
;;; If you mung this you might wish to mung CHTN similarly.
;;; skips unless the luser should be warned.
lsrchk: pushj p,winchk ;are they a winner?
skipa ;nope
jrst popj1 ;yea; so skip
;; check to see if they're in loser table
move t,[-loslen,,lostab]
;; winchk left xuname in tt
lsloop: camn tt,(t)
popj p, ;loser -> don't skip
aobjn t,lsloop
;; check inquir entry
movei a,filech
move b,[-lsrpgs,,hstpag];map lsr where host table will be
pushj p,lsrtns"lsrmap
jrst lsrlos ;something wrong; print loser message
.suset [.rxuname,,b] ;xuname into b
pushj p,lsrtns"lsrunm ;find our entry in database
jrst lsrlos ;no inquir entry; must be loser
.close filech, ;close lsr file
movei a,lsrtns"i$grp ;we want to check group
pushj p,lsrtns"lsritm
popj p, ;none means guest
ildb a,a ;get field
caie a,"A ; AI lab
cain a,"L ; LCS
jrst lsloo1
caie a,"V ; NIL group
cain a,"Z ; Clinical Decision Making
jrst lsloo1
caie a,"C ; Theory of Computation
popj p, ;they're not a lab member
lsloo1: movei a,lsrtns"i$rel ;Check relationship
pushj p,lsrtns"lsritm
popj p, ;none means loser
ildb a,a ;get field
caie a,"X ;some losers put AX when they mean XA
;; These names are for NETWRK
popj1: aos (p) ;otherwise skip
cpopj: popj p, ;when returning
lsrlos: .close filech, ;close lsr file
popj p, ;don't skip
;;; Skip means these people know what they're doing; don't hassle them
winchk: .suset [.rxuname,,tt]
move t,[-winlen,,wintab]
wnloop: camn tt,(t)
jrst popj1 ;winner; skip
aobjn t,wnloop
popj p,
;;; these people aren't hassled
wintab: irp x,,[gumby,alan,cstacy,cent,devon,dph,oaf,mly,jtw,jnc,klotz,cbf,sra,ed,swa,map,jbvb]
sixbit/x/
termin
winlen==.-wintab
;;; These people are always hassled
lostab: irp x,,[tk,hewitt,rms]
sixbit/x/
termin
loslen==.-lostab
;;; warn loser that net-hopping is a waste of the machine
lsrwrn: movei tt,[asciz/
Using this machine to access another will needlessly
drain resources, slowing the system both for you and other users.
You'll probably get better response by connecting directly.
You should not do this unless you have a good reason to do so. If
you have any questions, typing :LUSER to DDT will request a system
programmer to assist you.
Are you sure you want to net-hop?/]
pushj p,outstr
$call iot,[#ttyich,a],[#%tinwt] ;Wait for character
.lose %lssys
caie a,"y
cain a,"Y
popj p, ;it was a Y
caie a,40 ;maybe it was a space?
.logout 1,
popj p,
Great, but please include other vulnerabilities, including exploits, like mine in 1999 [1]:
In the xterm there is a feature to change the title of
the window. You can change the title of the window sending one of the escape codes of the
xterm.
(linux: man console_codes)
By Example:
ESC]2;This is my Xterm^G
This escape code changes the xterm's title to "This is my Xterm"
Obviously You can do the same using the kvt (Kde Virtual Terminal).
But the kvt has a buffer overflow. If the size of the new title of the
window is big enough then the kvt will do a core dump.
This bug follows the "reverse exploit" line, if some
program sends
this escape code to the kvt.
For Example, When someone connects to any ftp server and the server sends
the Welcome Message, It will be easy to exploit this bug changing the
Welcome Message (in the .message file) to one with this
escape code and
to cause a buffer overflow.
Another example where someone can cause a buffer overflow in your machine is
simply doing "cat hosts" where hosts may be a file that
you received
by mail containing the "change window escape code".
This bug shows some of the kvt's security problems being exploited via
a "reverse exploit" or a exploit sent directly to your
terminal
(if the attacker can write to your kvt)
If this bug is exploited, then the attacker can obtain
the privileges of the kvt's owner and execute some arbitrary code as this user.
[I found after] This bug was reported to the kde team by Larry Granroth
in January.
(http://bugs.kde.org/db/33/332.html)
The new kde's version doesn't have this bug in the
kconsole
Kvt was replaced totally by kconsole.
But the RedHat 6.0 installed with KDE has this bug.
Thanks, I will see about adding that. As I wrote before CVEs it is hard to find details. I've also noticed Google is missing quite a lot of historical things lately.
No it isn't. The problem is using regular expressions as the tool for doing it. By feeding the character stream through a proper DEC VT state machine, it's actually fairly trivial.
After all, terminal emulators do exactly this. And it's easy to spot the ones that use state machines instead of regex string matching (which, alas, a few do). The string matchers have had loads of bugs and problems over the years handling unknown but correctly formed control sequences, which the state machine implementations just swallow and ignore with ease.
DEC VT is far from the only escape code set. That's more my point. I agree regex isn't the best tool especially for stream processing, but the amount of escape codes out there (and the fact anything can be an escape code...) is what makes it "nearly impossible".
Disable, filter out ansi escape sequence(s), rewrite/map ansi escape sequence(s) to tput format.
No source code or familiarity with using LD_ tricks? use python wrapper around program and script python for appropriate mods before passing along to window.
Wonder if this was one reason for Sun Solaris NeWS (post script based vs, text based window manager)
Perhaps use python from command line with gui extensions and no ansi escape sequence support as window manger.
Run terminal window without ansi escape seqence support. Better coverage if done with ebpf filter at the OS level.
The OS has nothing to do with this aside setting the title I suppose, and that's if native widgets are being used and depends on the system.
Further, there's nothing special about escapes. Their "feature" is that they're just text, part of the normal stream of bytes. Anything can be an escape if it's interpreted as such by whatever is consuming the stream.
> "The OS has nothing to do with this aside setting ..."
note: Additional technical information context/solution provided for answer is a bit beyond the typical response required for question.
OS has better control/access to user set of i/o streams.
Not the most efficient/convenient way, but ebpf can be setup to detect 'escape text patterns' and filter by user/xwindows widgets/aspect of program such as title. Would be a pain, because of need to deal with inband/non-cannonical characters (ctrl-D & EOF being majority of live stream problem)
Sticking to 'title change' is a non-continuous stream aka for this talk, first non-connonical character closes the stream, aka no distinction between Form Feed / Line Feed or ctrl-D.
Simple user land way would be to redirect terminal to sed /awk script to identify 'title' command, strip out the 'security sequence issues' before passing along i/o to what ever is going to do the 'window with title to change'.
OS, windows manager, initial login shell and process initiating the window who's title are to be changed are parent processes with io points that can be redirect with the aforemention script.
in-band control for terminals also allows for redirecting to remote machines via programs like telenet without the need for ssh! (for security, would need to do a few other things too) 1st process to
Using loopback device/socket instead of named/anonymous pipe redirection will allow for raw/non-cannonical input (control done out of band) -- assuming the filter program (other than standard user shell) setup to handle/pass along raw/cannonical input/output.
Sort fun way to customize 'rm -rf' with a pop-up check question 'Are you really sure you want remove everything' and terminate the command if unintentional (and/or keep tabs on what removed by running find command / mail list).
further search terms: stty, tty, raw, cooked, canonical, non-cannonical
For this topic, perhaps confirm with user if really wanted to allow (escape sequence), retry(filter out ansi sequence), aboard(skip it). Or just log attempt for review/action later.
I hate inline escape sequences, but I hate ioctl/SetConsoleTextAttribute even more.
I also think that if someone (including, of course, the bumbling fool known as myself) ran something on my terminal I'm already toast even before the terminal misbehaves - because most sandboxes/mandatory access control systems are either a complete joke or disabled most of the time.
You don't even need to run something untrusted, just compromised console output is enough (from server logs, for example they mention python -m http.server)
Yes, and it's "even better" that your average RCE because it's not running code on the potentially sandboxed server, but on the (statistically speaking not sandboxed) developer/admin machine with potentially privileged access to other machines and systems.
Someone really ought to design a new text terminal protocol with a strict binary separation between data and control information and eliminate the regrettable scourge of escape sequences forever. Then using a command like "cat" to output a binary data to the screen would cause no unfortunate side effects under any conditions, a program would have to use a different interface to control the terminal, even to move the cursor from one place to another or to clear the screen. A proper terminal should not even process backspace characters unless they arrive via a control channel rather than the data channel, let alone process all sorts of other more dangerous control sequences.
Is there a network protocol that goes along with that? Something suitable for a text terminal? Not just a proprietary and more than likely underdocumented API?
Related to mosh (mentioned in the article), something has been bouncing around my head a lot. Why don't terminal emulators already work the way mosh does, decoupling the terminal part and the rendering part?
I assume it is possible to ingest a stream of terminal data much faster than rendering it (obviously keeping track of what is needed to render). Then the render loop can happen independently at a certain framerate, like a game engine. It is quite frustrating when a verbose 'cat' or 'find' slows my terminal emulator to a crawl drawing each line.
(edit: it just occurred to me that maybe this would break scroll-back in the terminal emulator... hmm)
In general you're better off using a terminal that performs better, because extra buffering would be annoying the other way around in the usual throughput/latency tradeoff (you'd press ^C and then it would continue to display what's in its buffer to you, rather than reacting quickly).
What mosh brings is decoupling the rendering across the network. A lot of the poor perceived performance over high-latency links happens because ssh puts your terminal into raw mode, so even if the line is being echoed back, that is going all the way to the remote system and back again.
It's actually possible to fix line editing in ssh, without using something like mosh, see for example https://github.com/hyc/OpenSSH-LINEMODE. It's a shame OpenSSH hasn't merged something like those (now quite old) patches.
I (vaguely) remember playing games with terminal echoback on physical terminals back in the early-mid 1980s when I was in college. This was on a VAX/VMS system.
Someone (I don't remember who did what here) discovered that they could get `SHOW SYSTEM` (roughly analogous to unix `ps` command) to display their name in reverse video by adding escape sequences to their process name. So a bunch of us started experimenting to see what else we could embed in there.
Most of the terminals attached to the VAX were Zenith Z-19s, which mostly emulated DEC VT-52s but with some added features. One of those added features was an enablable 25th line (in addition to the regular 24x80 display) that functioned as a sort of status line. We found we could enable that, write something into it, then use the "transmit 25th line" escape sequence to send its contents back to the VAX. I remember having to work around limitations like it sending an escape sequence before the 25th line (which confused VMS), and I think it didn't send a carriage return at the end... or something like that.
I don't think we ever got it to do anything terribly interesting, but it was fun to play with. And then IIRC a VMS update blocked control characters in the `SHOW SYSTEM` listing.
Which is why a 2003 paper being "the first write-up I can find of the potential issues with terminal security" doesn't sit well. Quite a lot of this stuff was written up decades ago. Just on Usenet and BBSes and documents that circulated on FidoNet, before the World Wide Web even existed.
I wrote a VT100 (+ Sun extensions) emulator in PostScript that had a magical "^_PostScript Code^]" escape sequence that let you download arbitrary PostScript code into the NeWS window server (which could do anything you want)! Also ^A would print out the stack, because why not? ;)
It also supported the <ESC>#8 "fill screen with E" test sequence, just so it could pass VTTest, plus a few extra Sun and AAA sequences that my slave driving boss Hugh Daniel demanded (since he was from Ann Arbor). Yes, a company actually called "Wedge Computer, Inc." paid me to write it!
There are also some subtleties about missing numeric arguments that VTTest checks. My SoftTTY emulator would detect syntax errors in escape sequences, which implied you were trying to test it with VTtest, so then it would change the cursor to the shape of a hand giving you the finger (shape-error-cursor), and flash it 20 times, to flip Hugh off for subjecting my terminal emulator to VTTest.
/ps-string-prefix {
/ps-string 256 string def
/ps-string-len 0 def
/termulate-char //termulate-ps-string def
} def
/termulate-ps-string {
dup 27 eq {
pop
reset-termulator
ps-string 0 ps-string-len getinterval cvx
{ exec } stopped pop
} {
ps-string ps-string-len 3 -1 roll put
/ps-string-len ps-string-len 1 add def
ps-string-len ps-string length eq {
/ps-string
ps-string-len dup add string
ps-string 1 index copy pop
def
} if
} ifelse
} def
I used it for writing a Lisp "Little Backwards Chainer" program that would download a tree of dirty knowledge base symbols into the server to define a corresponding tree of pop-up pie menus, so I could browse the Lisp data structures as pie menus to debug the program.
; Write dirty nodes to the NeWS server in PostScript, and mark them as
; clean.
(defun sweep ()
(clean-tree '*KB*))
(defun clean-tree (node)
(cond ((get node 'dirty)
(send-node node)
(putprop node nil 'dirty)))
(cond
((null (node-alist node)) nil)
(t (mapcar #'(lambda (link)
(clean-tree (link-node link)))
(node-alist node))
nil)))
(defun send-node (node)
(ps-make-node node (node-sexp node)
(node-associated-atom node)
(node-alist node)))
; Magic control characters the PostScript terminal emulator responds to.
(setq StartPS "^_") ; Ctrl-_: begin trojan horse
(setq EndPS "^]") ; Escape: end and execute trojan horse
; Write out a NeWS menu definition, with the right control characters to
; make the PostScript terminal program (SoftTTY) execute it.
(defun ps-make-node (name sexp associated-atom alist)
(msg "% Sending " name "...")
(msg StartPS "userdict /" name " [" N)
(msg " " (list 'Name= name) " {" (ncons name) " print}" N
" " (list 'Sexp= sexp) " {" (ncons sexp) " print}" N
" " (list 'Atom= (fact-name associated-atom))
" {" (ncons (fact-name associated-atom)) " print}" N)
(mapcar #'(lambda (link)
(msg " " link " /" (link-node link) " cvx" N))
alist)
(msg "] /new DefaultMenu send put" N)
(msg name " /Clockwise false put" N)
(cond ((eq name '*KB*) (msg "/ClientMenu *KB* def" N))) ; kludge
(msg EndPS N))
Hugh was the author of ColorToy and ColorTool, which lets you play with and change all the user interface colors (even to dynamic random procedural colors that changed every time it paints, which was the most fun, and useful for debugging drawing, incremental screen updating, and clipping bugs!)
Thanks! Some of the best stuff is hard to find. I just had ChatGPT analyze the code, describe the rules, and explain the cultural references in an OPS-5 program I wrote in that same AI class (from Jim Hendler), in the aftermath of the Morris Worm and Iran Contra Scandal.
It is something that I had considered too. (For this reason, some of my shell scripts that involve curl (for sprunge and for icanhazip) have "| cat -v".)
Ideally there should be some way to cancel responses (echoback), which would avoid some of the problems; I am not sure if this can be done. Another thing I had considered is to make a "universal escape sequence" which is recognized anywhere including in the middle of something else. There are other considerations, too.
(I had considered writing a terminal emulator, and will have to consider this and other things.)
(I had also considered operating system design, where the command-line interface would work differently than this anyways.)
> VTE based terminals, Kitty and WezTerm are some of the few terminals that accept C1 controls by default, within UTF-8 encoded data. My recommendation would be to not use these terminals with untrusted data.
It seems the author didn't look at the code because in VTE, bytes enter UTF-8 decoder first and therefore the parser is already dealing with 32-bit unichars.
I agree the only correct way C1 controls can work is encoded within UTF-8 data, else nothing works.
The context is escaping C0 control characters is simple, you look for a single byte and filter it as you need. C1 controls when not encoded as UTF-8 are also single byte characters and therefore easy to filter out. For multi-byte encodings, you need to correctly decode the encoding, then filter, this has synchronisation problems and is tricky[1] (particularly as a Unix byte stream doesn't tell you it's encoding, although you can assume if someone is trying to display it as text it is probably UTF-8 these days).
I should probably expand on that recommendation more, there's a lot in the paper, but the crux of the issue is C1 controls are a legacy thing and serve no useful purpose anymore and as I've shown there are enough issues just dealing with C0 controls.
If C1 controls are encoded as UTF-8 as is the only way they can work on a modern system, then they take up as many bytes as C0 controls (e.g. CSI is "\e[" or U+009B, which encoded as UTF-8 is 0xC2 0x9B) so they don't even save bytes on the wire.
I am waiting for a Vim exploit in a similar way.. I map insert mode jj and kk to escape to leave insert mode. Sometimes I paste text in insert mode and it's interpreted literally in Vim, so I am waiting for some exploit like modifying a clipboard to be like:
I have a binding to toggle paste mode. It would prevent the attack, but it also disabled jj/kk insert mode bindings... I do try to use the toggle before pasting, but I'm human and often forget.
Seriously, it’s the only safe way to paste into a terminal. When you initiate a paste your terminal emulator sends a control sequence that tells the program that a paste is happening so that it won’t interpret any of it as anything but text to be inserted. It’s completely automatic. And your terminal emulator supports it! So does vim!
It’s only be available in xterm for 21 or so years, so I guess it’s not _that_ surprising to find that people still don’t know about it.
Bracketed paste mode is good and ideally it just works(tm) and therefore people don't need to know about it. It does just work in recent versions bash (readline) and Zsh.
Unfortunately there is still poor handling of it in some terminals. I posted[1] yesterday on oss-security that ^C for example can be placed on the clipboard and may end bracketed paste mode in some circumstances.
For xterm that can be mitigated by adding "ETX" to disallowedPasteControls[2]:
That still isn't enough (e.g. ^\ or ^Z can be used too in some contexts). Other terminals are looking at this currently (e.g. kitty[3]). I suspect there will be some other patches coming out.
Even things like the use of ESC, and the overloading of ESC for the actual ESC key, for a signal for keycodes, and for the ALT/META key are crazy.
I've thought for a while that I should implement an extremely reduced set of escape sequences which use a non-ESC character (maybe SOH or something) for control sequences and key encoding. A minimal set -- color / cursor position / save-screen / query-size -- whatever it takes to make vim/emacs + tmux/screen work, and call it a day.