Things you didn't know about GNU readline (2019)

Along the same lines, I find this little utility in standard linux repos to be amazing. Try it on ed: `rlwrap ed`.

  Package: rlwrap
  [...]
  Homepage: https://github.com/hanslub42/rlwrap
  Description-en: readline feature command line wrapper
    This package provides a small utility that uses the GNU
    readline library to allow the editing of keyboard input 
    for any other command.  Input history is remembered 
    across invocations, separately for each command; history 
    completion and search work as in bash and completion 
    word lists can be specified on the command line.

Some years ago I helped diagnose an O(n²) bug in readline.

For years, I had problems pasting many lines of SQL in a psql session. It would slow down as it worked through the characters. One day the same happened in another shell, and it dawned on my that this was probably a problem with readline.

So I fired up a profiler and sure enough, something inside readline was recalculating something upon each character input with the work done proportional to all the characters already input.

I sent my findings to the mailing list, and Chet Ramey fixed it.

In the process, I discovered that wonky editing of long lines when resizing a terminal window was also fixable. When you resize, you change the line width, so readline needs to take that into account when it wraps the long lines. I think it did have code to do that at the time, but it just didn't kick in for some reason. Can't remember the details, but it does seem to work fine in psql today. That was another multi-year annoyance.

It's a relief seeing readline pulled in as a dependency for some command line tool: ahhh, there's going to be at least a reasonable amount of command history and editing features available.

To remind yourself how much we take for granted, play with `psql --no-readline` some time and see how awful it is to lose the ability to up-arrow get the last query back, edit it, and send it again.

^U in readline isn't an emacs command (that keystroke does something else in emacs); it's the old teletype protocol for line input.

So if you aren't using readline, ^U deletes everything back to the beginning of the line. Readline preserves that because us pre-readline old farts have it wired into our fingers.

Before display terminals there were just printing terminals: IIRC # deleted a character and @ everything to the beginning of the line. This goes back to Multics and was carried forward in early releases of Unix at Bell Labs.

This is obviously pretty inconvenient when you have a display terminal but I am pretty sure that code remains in the bowels of the BSD TTY IO system at least and can probably be enabled. Maybe it even survived into linux for some compatibility reasons.

Also I WUZ THERE and I think Chet has it backwards on the POSIX controls issue: bfox talked about the TWENEX line input JSYS that had that stuff built in (I believe it was also, perhaps even earlier, in VMS, and maybe TENEX, and surely he had talked to some of those folks too)

Debian dash is a minimal POSIX shell that also is able to implement "set -o vi" (as suggested by POSIX.2).

http://gondor.apana.org.au/~herbert/dash/

The dash shell doesn't use GNU Readline to implement "set -o vi" but instead uses "libedit":

https://thrysoee.dk/editline/

Debian does not appear to link dash with libedit, so command line editing is disabled.

They should consider doing so, and offering dash as an interactive shell. This would give people fewer bashisms to unlearn when they need to write portable scripts for the POSIX shell.

The dash shell is actually reasonably pleasant for interactive use when compiled with libedit.

This is cool, I didn't actually know you could configure Readline so the title is apt for me :D

Much like in the article I'm also a Vim user who excited to try Vim mode at the command line and I almost instantly hated it. I suppose it's almost required to mention tpope's rsi.vim[0] which gives you Readline bindings in insert and command modes. I've adopted Readline everywhere in my OS except in Vim normal mode which I believe is not too uncommon for many folks.

[0] https://github.com/tpope/vim-rsi

> Also, without some sort of indicator, Vim’s modal design is awkward here—it’s very easy to forget which mode you’re in.

That is why I have this in inputrc:

set editing-mode vi

set show-mode-in-prompt on

set vi-ins-mode-string +

set vi-cmd-mode-string :

"Tens of thousands of people probably use it every day without thinking about it."

Probably tens of millions at this point.

One important characteristic of this `readline(const char*)` call is that it blocks. So, if you're trying to write a REPL for a language that uses green threads for concurrency - I'm looking at you, GNU Smalltalk - then DON'T. At least not from the main thread.

> Also, without some sort of indicator, Vim’s modal design is awkward here—it’s very easy to forget which mode you’re in.

With `bash`, you can show which mode you're in by putting this in your `~/.inputrc`:

    show-mode-in-prompt on

It makes your prompt look like this:

    [foo@bar ~]$ # before turning it on
    @[foo@bar ~]$ # after
    @[foo@bar ~]$ set -o vi
    (ins)[foo@bar ~]$ # and after I press esc:
    (cmd)[foo@bar ~]$

Customize these with `emacs-mode-string`, `vi-ins-mode-string`, and `vi-cmd-mode-string`.

Bash uses readline, right? So

> Tens of thousands of people probably use it every day without thinking about it.

seems like an understatement.

> One thing I’ve done is reconfigured Ctrl-K. Normally it deletes from the cursor to the end of the line, but I rarely do that.

I would recommend against this. I delete from the cursor to the end of the line all the time. Maybe more in Emacs than readline, but still

You can already delete the whole line by issuing C-a then C-k. I also do this all the time and that key combo is as easy for me to type as any two letter word.

Remapping C-k means you now have two ways to kill the whole line and no way to kill from the cursor.

But worse: you're teaching yourself a keyboard setup that only you use. You'll be lost on somebody else's computer, or any time you don't have your config installed. The standard readline bindings turn up in sometimes surprising places and it's a joy when you realise you're already trained to use some system or software you've never used before.

I'm surprised the article doesn't also mention C-y. After you've C-k killed something it goes on to the "kill ring". Press C-y to bring it back! (You can bring more than the most recent thing back, but you now have the key words to look this up yourself).

Lovely opening paragraph :

> I sometimes think of my computer as a very large house. I visit this house every day and know most of the rooms on the ground floor, but there are bedrooms I’ve never been in, closets I haven’t opened, nooks and crannies that I’ve never explored. I feel compelled to learn more about my computer the same way anyone would feel compelled to see a room they had never visited in their own home.

This is nostalgic. For a Systems Programming course, I had the following assignment:

- Implement readline(3)

- If you have any questions, see the man page.

- Extra credit: implement all GNU extensions.

To make Less (the pager program) behave a bit more as if it used Readline (it doesn't), one can create a `~/.config/lesskey` file with these contents:

  #line-edit
  ^A home
  ^E end
  ^F right
  ^B left
  ^P up
  ^N down
  ^D delete
  ^W word-backspace
  \ed word-delete
  \ef word-right

I think a relatively recent version of Less is required.

Ctrl-k kills the rest of the line. Ctrl-u kills the entire line.

Ctrl-y yanks stuff back in from your kill ring. Esc-y will then replace the stuff with the previous item in the kill ring.

Ctrl-arrows moves a word at a time.

Ctrl-r starts a reverse interactive search through the history. Use any editing key, like ctrl-e to start working on the current search result.

One thing trying to work with Guix over the last five years taught me is how much I take for granted all the little things that come packaged extra. I was using guix for about a month before I learned that readline was the package that allowed me to do a lot of the extra commands like ctrl-c etc.

Some more good ideas for Readline configuration: https://www.topbug.net/blog/2017/07/31/inputrc-for-humans/

The article feels incomplete for not not mentioning the bash built-in called bind. It allows you to work with the reeadline bindings.

I use Ctrl-R a lot to find previous commands from my shell history. Often I have to press Ctrl-R several or even many times to find the right one. And then it happens that I overshoot and would like to change the search direction. Unfortunately forward-search-history is on Ctrl-S, which is overloaded by flow control. Occasionally I use flow control, so I cannot remove that. What solution are others using to get forward-search-history?

Shout-out to the clink project, which brings a modern shell experience (including Readline) to Windows' cmd.exe. This is the most advanced shell I have across Mac/Linux/Windows, which is always a bit surprising. It's extremely extensible and has some pretty incredible functionality written for it (also by the maintainer).

https://github.com/chrisant996/clink

Some years ago I sent a bug report to the email listed at the end of a man page.

I don't recall what the bug was, I also don't recall why the video I sent showcasing the bug had my national anthem in the background. But it sparked a nice bug report exchange.

That he is doing this for no pay and for the passion of providing software to users all around the world fits.

Remiss for not mentioning "rlwrap": https://github.com/hanslub42/rlwrap

...basically "readline in a can" that you can use to improve text input of other programs.

I forget if it was RMS or ESR that really wanted to hold "readline" away from corporations, so it's licensed as GPLv3 (not v2), and he explicitly wanted it to _not_ be LGPL (ie: embeddable).

My crazy readline/bash tip is `ESC, .` (or maybe `alt+.`, I'm not 100% sure, random search result here: https://gist.github.com/tuxfight3r/60051ac67c5f0445efee ). Basically it up-arrows through the last "word" of previous commands, adding it to the current command.

Example:

     mkdir foo
     cd `esc, .` => cd foo
     vim bar.txt
     cp `esc, .` baz.txt => cp bar.txt baz.txt

It's crucial in that it reduces the number of keystrokes, and avoids typing errors completely, as you're "recalling" something rather than re-typing.

I'm also in the same boat as the author in that `set -o vi` doesn't quite do it for me (although I consider myself an expert in vi). What I tend to do instead is:

     C-a # <cr>   => go to the beginning of the line and insert a comment char
     fc           => "fix command", see "help fc" (it's a built-in)

...basically it invokes: `$EDITOR "$( history | tail -1 )"` and whatever you save/quit will get executed as a command.

This lets me intentionally "damage" a command, but then get into `vim` to "fix it up" (eg: `dw, cw, df/, etc...`). Remove the comment from the beginning of the line to execute it, or keep it there to take the command back "out" of `vim` so you can keep building it up.

I was writing a command-line tool in Python, and wanted bash-like line editing capabilities. And then I realized that there was a Python wrapper for readline. Problem solved!

Great thing, but it is GPL... there is libedit that's LGPL and while isn't a complete replacement, is good enough.

Readline is good, but once I started using the xonsh shell, and prompt_toolkit, going back to Readline always feels a bit lacking.

  > Ramey has now worked on Bash and Readline for well over a decade. He has never once been compensated for his work

I just want to remind everyone of the xz hack. The lesson here is that a few people maintain projects that are critical to our infrastructure. You may be familiar with the XKCD comic[0], but this makes it look like the unpaid single/few maintainer critical projects are rare.

These people are generally unpaid and that has big ramifications. Sure, it makes them susceptible to bribes and more prone to hostile takeovers (however you view xz), but there's more. It means these works are simply a work of passion, and they are working on them in their spare time. We all know here that that time is less and has lower priority than our work time. So you're more likely to be rushed and make mistakes, or not be able to fix things fast enough, and so on. Is this really the way we want this infrastructure to be maintained? Startups have more stability than this and are more likely to pass the "CEO hit by a bus" rule[1]. Not to mention the difficulties of transitioning when the dev wants to retire or move onto something else. This does lead to issues, and while we've solved them in the past, that doesn't mean we can't do better.

Surely they deserve some compensation. We have 8 companies which are worth over a trillion dollars and certainly all 8 use readline. Are you telling me that these companies can't collectively kick back $200k/yr. That's literally what Elon makes in 10 minutes if he gets 5% on his wealth. I find it hard to believe we don't have enough money to fund possibly thousands of these projects (if there are that many, but there are likely that many people). It only takes Ballmer 16 minutes and there are 20 people who can do it in 30 minutes or less (~50 for 1hr). Collectively the top 10 make 230 million per day for doing nothing. I think we have the money. (remember, we're talking about products that these companies are using. This isn't just a "pure donation," as these things have clear market value and the loss of the maintenance will result in these companies also losing money)

At the least, I'd like to remind everyone that the company you work for likely offers some matching donation. If you're at one of the big tech, they do.

[0] https://xkcd.com/2347/

[1] If your CEO, or any single specific employee, is hit by a bus and can no longer work, does the entire company also collapse?

IME

   socat readline prog  

always feels faster than

   rlwrap prog

This is way off topic but 2 hours ago I was trying to figure out how to read a single line from stdio using Rust/tokio and failed to figure it out in a reasonable time. It was infuriating.

That's nice, but if GNU readline disappeared from my Linux install I probably wouldn't notice.