SSH, HTTPS, Keys, Tokens and Scopes

42.0 What this chapter gives you#

1. You will be able to explain the two ways git talks to a server, SSH and HTTPS, and say exactly what is different and what is identical.
2. You will be able to say what SSH is, who wrote it, in what year, and what dangerous older tools it was built to replace.
3. You will be able to explain a key pair in plain words and then precisely, including what the private key can do that the public key cannot.
4. You will be able to read a real ssh-keygen run, decode the public key file field by field, and choose between RSA, ECDSA and Ed25519 with reasons.
5. You will be able to explain authorized_keys and known_hosts, including why wrong file permissions break logins silently.
6. You will be able to explain why the reader’s SSH on port 22 hung while HTTPS on port 443 worked, and apply both fixes.
7. You will be able to explain personal access tokens, their expiry, their prefixes, and why a leaked token is an emergency rather than an annoyance.
8. You will be able to explain a scope precisely, and explain the exact reason a push touching a file under .github/workflows/ was refused.
9. You will be able to tell a permission failure from a network failure from a rejected push, even though all three print “push failed”.
10. You will be able to diagnose at least eight real git authentication error strings from the text alone, and act on each one correctly.

42.1 The two transports git uses, and why there are two#

PLAIN42.1.1 in simple words#

1. Git keeps your project history on your own machine. All of it.
2. Sometimes you want to send that history to a server, or fetch new history from one. That is what push and fetch do.
3. To send anything over a network, two programs need a pipe between them.
4. Git does not build its own pipe. It borrows one that already exists.
5. There are two pipes it commonly borrows, and you choose which one by the shape of the address you type.
6. An address like git@github.com:owner/repo.git uses the SSH pipe.
7. An address like https://github.com/owner/repo.git uses the HTTPS pipe, the same kind of pipe your web browser uses.
8. Here is the part people get wrong. The repository on the server is the same repository in both cases. One box, one set of files.
9. Only the road to it changes. It is like posting a letter versus handing it over in person. Same letter, same house, different delivery.
10. So switching from one to the other never loses a commit, never rewrites history, and never needs a re-clone. It is a one-line change.
11. What does change is how the server works out who you are.
12. Over SSH you prove yourself with a key file on your laptop.
13. Over HTTPS you prove yourself with a token, which is a long secret string that behaves like a password with rules attached.

PLAIN42.1.2 a picture in your head#

1. Imagine one library building with one collection of books.
2. There are two doors. The staff door at the back and the public door at the front. Both lead to exactly the same shelves.
3. The staff door has a lock. You carry a physical key that fits it. If your key fits, you walk in. Nobody asks your name.
4. The public door has a desk. You show a card with your name on it and a list of what you are allowed to do. The clerk reads the card and decides.
5. The books do not care which door you used. A book you borrow through the back door is the same book you would have borrowed through the front.
6. Now suppose the alley behind the building is closed for roadworks. The staff door is fine. You simply cannot reach it.
7. You walk round to the front and use the public door. Nothing about the library changed. Your route changed.
8. That is exactly the reader’s situation in this chapter, and exactly the fix.

Where this comparison breaks:

9. A real key and a real card are both things you hand over. A private SSH key is never handed over. It stays in your pocket and answers a question.
10. And a library clerk can see your whole card. A well-built server only sees that your token is valid and what it is allowed to do, not your password.

PLAIN42.1.3 a worked example#

1. Here is the same repository addressed both ways, on a real machine.

$ git remote add t git@github.com:octocat/Hello-World.git
$ git remote add t https://github.com/octocat/Hello-World.git
$ git remote add t \
    ssh://git@ssh.github.com:443/octocat/Hello-World.git

2. All three name one repository: the account octocat, the repository Hello-World.
3. The first is the short SSH form. The colon after the host name is not a port. It separates the host from the path.
4. The second is HTTPS, and it is an ordinary web address with a .git suffix.
5. The third is the long SSH form, written as a proper address with a scheme, which is the only form where you can write a port number.
6. Changing transport on an existing clone is one command:

$ git remote set-url origin https://github.com/owner/repo.git
$ git remote -v
origin  https://github.com/owner/repo.git (fetch)
origin  https://github.com/owner/repo.git (push)

7. Nothing else changes. Your branches, your commits and your stash are all still local files under .git, untouched.
8. Here is a real clone of a real public repository over HTTPS, from the machine used to write this chapter:

$ git clone https://github.com/octocat/Hello-World.git
Cloning into 'Hello-World'...
$ git log --oneline -1
7fd1a60 Merge pull request #6 from Spaceghost/patch-1

9. Over SSH the commit 7fd1a60 would be byte-identical, because a commit identity is a hash of its content, not of how it travelled.

PLAIN42.1.4 what is really happening inside#

1. Git itself does not know how to speak to a network. It shells out.
2. When you give it an HTTPS address it runs a small helper program called git-remote-https. When you give it an SSH address it runs ssh.
3. You can watch it choose. Here is a real trace on the machine used for this chapter, with the timestamps trimmed for width:

$ GIT_TRACE=1 git ls-remote \
    https://github.com/octocat/Hello-World.git
trace: built-in: git ls-remote https://github.com/...
trace: run_command: git remote-https https://github.com/...
trace: exec: git-remote-https https://github.com/...

4. And here is what git runs for an SSH address. This is the real command line git built, wrapped for width:

$ GIT_TRACE=1 git ls-remote \
    ssh://git@ssh.github.com:443/octocat/Hello-World.git
trace: run_command: GIT_PROTOCOL=version=2 ssh
  -o SendEnv=GIT_PROTOCOL -p 443 git@ssh.github.com
  'git-upload-pack '\''/octocat/Hello-World.git'\'''

5. Read that last line carefully, because it explains everything about SSH and git in one go.
6. Git asked SSH to log in as the user git on the host ssh.github.com, and then to run one command there: git-upload-pack.
7. SSH is not doing anything git-specific. It is doing what it always does: logging in and running a command on a remote machine.
8. The remote command’s input and output are piped straight back down the encrypted connection to your local git.
9. Over HTTPS there is no login shell and no remote command. Instead git makes ordinary web requests. Here are the real ones, captured live:

GET /octocat/Hello-World.git/info/refs?service=git-upload-pack
Git-Protocol: version=2
<= HTTP/1.1 200 OK
<= Content-Type: application/x-git-upload-pack-advertisement

POST /octocat/Hello-World.git/git-upload-pack
Content-Type: application/x-git-upload-pack-request
<= HTTP/1.1 200 OK
<= Content-Type: application/x-git-upload-pack-result

10. One GET to ask what branches exist, then one POST carrying the real conversation. That is the whole thing.
11. Underneath both, the same objects flow: the same commits, trees, blobs and the same packfile format. The payload is identical.

TECHNICAL42.1.5 the engineer’s version#

1. Git supports four transports. Two matter today.

2. GitHub removed support for the unencrypted git:// protocol as part of a protocol hardening programme that became permanent on 15 March 2022.
3. Over SSH, git uses the SSH connection protocol to request a single command execution, not an interactive shell. The remote side runs git-upload-pack for fetches and git-receive-pack for pushes.
4. The environment variable GIT_PROTOCOL=version=2 is passed with -o SendEnv=GIT_PROTOCOL. Protocol v2 was introduced in Git 2.18 in June 2018 and made the default for fetches in Git 2.26 in March 2020.
5. Protocol v2 matters on large repositories because it lets the client ask for a filtered reference advertisement instead of receiving every ref.
6. Over HTTPS, git speaks the smart HTTP protocol, defined in the Git documentation file http-protocol.txt. Two endpoints per operation: GET $URL/info/refs?service=git-upload-pack then POST $URL/git-upload-pack.
7. The media types are application/x-git-upload-pack-request and application/x-git-upload-pack-result, and the push equivalents replace upload with receive.
8. Rewriting one transport into the other is a supported config feature, and it is the cleanest fix for a blocked port:

$ git config --global \
    url."https://github.com/".insteadOf "git@github.com:"
$ git config --global \
    url."https://github.com/".insteadOf "ssh://git@github.com/"

9. On the machine used for this chapter that rewrite is active, and it is visible in a real check:

$ git remote add t git@github.com:octocat/Hello-World.git
$ git remote get-url t
https://github.com/octocat/Hello-World.git

10. That is worth knowing as a trap. If a colleague swears their SSH URL works on a machine where port 22 is blocked, check insteadOf before believing that port 22 is open.
11. pushInsteadOf is the write-only variant, letting you fetch anonymously over HTTPS while pushing over SSH.

WORDS42.1.6 remember these#

Transport — the road the data takes — the protocol carrying the Git wire protocol, SSH or HTTPS.

Remote — a nickname for a server copy — a named URL plus fetch and push refspecs stored in .git/config.

git-upload-pack — the program that sends you history — the server-side process serving fetch and clone.

git-receive-pack — the program that accepts your history — the server-side process serving push, and the one that runs the hooks that can refuse you.

Smart HTTP — git talking over web requests — the info/refs plus git-upload-pack request pair using x-git-* media types.

insteadOf — a rule that rewrites addresses — a url.<base>.insteadOf config key that substitutes URL prefixes before the transport is chosen.

42.2 What SSH actually is#

PLAIN42.2.1 in simple words#

1. SSH stands for Secure Shell. A shell is the text prompt where you type commands.
2. SSH lets you get a shell on a computer that is somewhere else, over a network, safely.
3. “Safely” means two separate promises, and both matter.
4. First, nobody in the middle can read what you type or what comes back. It is scrambled on the wire.
5. Second, you can be sure the machine you reached is the machine you meant. Not an impostor sitting in the middle.
6. Before SSH, people used tools called telnet, rlogin and rsh to do the same job. Those tools sent your password as plain readable text.
7. Anyone able to watch the network could simply read it. Not crack it. Read it, like reading a postcard.
8. In 1995 a researcher at a Finnish university discovered exactly that happening on his own network and wrote SSH in response.
9. Today SSH does far more than shells. It carries file copies, tunnels, and the git pushes in this chapter.
10. But underneath it is still the same thing: a safe way to run one command on another machine.

PLAIN42.2.2 a picture in your head#

1. Think of the old way as shouting your order across a crowded market square.
2. The stallholder hears you. So does everyone else, including the person who writes down what people order and comes back later pretending to be you.
3. Now think of SSH as a sealed pneumatic tube running from your desk to the stall.
4. You put a written note in a capsule, it travels inside the tube, and only the stall can open it. Nobody in the square sees anything.
5. But a tube alone is not enough. What if somebody quietly re-routed your tube to a different stall last night?
6. So SSH adds a second step. Before you send anything, the stall shows you a seal that only that stall could have made.
7. You compare it against the seal you wrote down the first time you visited. If it matches, it is really them.
8. If it does not match, everything stops with a very loud warning.

Where this comparison breaks:

9. A pneumatic tube is a physical private path. SSH runs over the same shared public cables as everything else. The privacy is mathematical, not physical.
10. And the “seal you wrote down the first time” is a real weakness. On that first visit you had no way to check. You simply trusted. We will look straight at that in section 42.5.

PLAIN42.2.3 a worked example#

1. A connection has three visible stages. Here is a real one against a small server, with the client asked to explain itself.

$ ssh -v -p 2222 root@127.0.0.1 'echo OK'
debug1: Authenticating to 127.0.0.1:2222 as 'root'
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher:
  chacha20-poly1305@openssh.com MAC: <implicit>
debug1: Server host key: ssh-ed25519
  SHA256:fwKYxD4JEmBcHpHj3WkgxSLrRh6tXIzpi/KOxz2yNrA
debug1: Host '[127.0.0.1]:2222' is known and matches the
  ED25519 host key.
debug1: Authentications that can continue: publickey
debug1: Offering public key: id_ed25519 ED25519
  SHA256:WH1E2gSLhG6FgMsha//GINoqouvX3fLxFRdMA3KMrSU
debug1: Server accepts key: id_ed25519 ED25519
Authenticated to 127.0.0.1 using "publickey".
OK

2. Line by line, in order of what happened.
3. kex is short for key exchange. Both sides agreed on the maths they would use to create a shared secret. That is stage one.
4. Server host key is the server proving who it is. Host is known and matches means the client checked its own notes and was satisfied.
5. Authentications that can continue: publickey is the server saying which ways of proving yourself it will accept. Here, only keys.
6. Offering public key then Server accepts key is stage two, you proving who you are.
7. OK at the end is stage three, the actual work happening inside the now trusted, now encrypted connection.
8. Notice the order. Encryption is set up first, then the server proves itself, then you prove yourself. Your credentials never travel in the open.

PLAIN42.2.4 what is really happening inside#

1. SSH version 2 is built as three stacked layers. Each has one job.
2. Layer one, transport. It runs immediately after the TCP connection opens. It has three tasks.
3. It agrees the algorithms both sides will use, it performs a key exchange to create a shared secret, and it verifies the server’s host key.
4. From that point on, everything is encrypted and every message carries an integrity check so tampering is detected.
5. Importantly, the shared secret is created fresh for this connection. It is never sent over the wire, by either side. Both sides compute it.
6. Layer two, user authentication. Now that the pipe is safe, you prove who you are. Public key, password, keyboard-interactive, or host-based.
7. This layer runs inside the encrypted tunnel, which is why a password sent over SSH is not exposed the way a telnet password was.
8. Layer three, connection. One SSH connection can carry many independent streams, called channels, at once.
9. A shell is a channel. A file copy is a channel. A forwarded port is a channel. They share the one encrypted pipe.
10. That is how you can be editing a file over SSH while a port forward runs in the background, on one connection and one login.
11. For git, exactly one channel is opened, of type “exec”, carrying one command: git-upload-pack or git-receive-pack.

   TCP connect (port 22)
        |
        v
 [1] TRANSPORT  algorithm agreement, key exchange,
                server host key check -> encrypted
        |
        v
 [2] USER AUTH  publickey / password / etc.
        |
        v
 [3] CONNECTION channels: shell, exec, sftp, forwards
        |
        v
   git-upload-pack runs on the server

TECHNICAL42.2.5 the engineer’s version#

1. SSH was written by Tatu Ylonen at Helsinki University of Technology in 1995, after a password-sniffing attack on the university network.
2. He released the first version in July 1995, and founded SSH Communications Security in December 1995 to commercialize it.
3. The protocol it replaced was the Berkeley r-commands: rlogin (TCP 513), rsh (TCP 514) and also telnet (TCP 23), all of which sent credentials in clear text.
4. Those tools also trusted a file called .rhosts, which authenticated by source IP address alone. Address spoofing therefore meant login.
5. SSH-1 and SSH-2 are different protocols, not versions of one protocol. SSH-1 has a structural integrity flaw.
6. SSH-1 used CRC-32 for integrity, which is a checksum, not a cryptographic authentication code. In 1998 this was shown to allow an insertion attack, and in 2001 the widely reported “SSH CRC-32 compensation attack detector” vulnerability made it worse. SSH-1 is dead and should never be enabled.
7. SSH-2 is specified in a set of IETF documents published in January 2006: RFC 4250 assigned numbers, RFC 4251 architecture, RFC 4252 authentication, RFC 4253 transport, RFC 4254 connection.
8. OpenSSH is a separate implementation, from the OpenBSD project. The history is precise and worth knowing.

9. The fork happened because releases after ssh 1.2.12 carried increasingly restrictive licence terms. Bjorn Gronvall’s OSSH revived the last freely licensed release, and the OpenBSD team took it from there.
10. SSH-2 support was written by Markus Friedl and shipped in OpenSSH 2.0 with OpenBSD 2.7 on 15 June 2000.
11. Port 22 is the IANA-assigned port for SSH. Ylonen requested it because it sat unused between ftp on 21 and telnet on 23.
12. Modern algorithm defaults on the OpenSSH 9.6 used for this chapter: key exchange sntrup761x25519-sha512@openssh.com, which combines the post-quantum lattice scheme NTRU Prime with X25519, and cipher chacha20-poly1305@openssh.com.
13. That hybrid key exchange became the OpenSSH default in version 9.0, in April 2022. The reason given was harvest-now-decrypt-later: an attacker recording traffic today and decrypting it after quantum computers arrive.

WORDS42.2.6 remember these#

SSH — a safe way to use a distant computer — the Secure Shell protocol suite specified in RFC 4251 through RFC 4254.

Key exchange (kex) — the step where both sides build a shared secret — a Diffie-Hellman style agreement producing session keys without transmitting them.

Host key — the server’s own identity key — the long-term key pair a server uses to sign the key exchange and prove it is itself.

Channel — one stream inside the connection — a multiplexed logical stream in the SSH connection protocol, with independent flow control.

telnet — an old unsafe remote login tool — a clear-text remote terminal protocol on TCP 23, RFC 854, with no encryption or integrity.

OpenSSH — the SSH program almost everyone runs — the OpenBSD implementation, first released 1 December 1999 with OpenBSD 2.6.

42.3 Public key cryptography for SSH#

PLAIN42.3.1 in simple words#

1. A key pair is two files made at the same time, mathematically joined.
2. One is called the private key. It stays on your machine and you never send it anywhere, ever.
3. The other is called the public key. It is meant to be handed out. You paste it into web forms and email it to people.
4. The private key can do one special thing: it can make a signature on a piece of data.
5. The public key cannot make a signature. It can only check one, and answer yes or no.
6. That asymmetry is the whole trick. Publishing the public key gives an attacker nothing, because checking signatures is not making them.
7. So when GitHub wants to know you are you, it does not ask for a password.
8. It sends your computer a piece of data and says: sign this.
9. Your computer signs it with the private key. GitHub checks the signature with the public key it already stored under your account.
10. If the check passes, only the holder of that private key could have produced it. You are in.
11. Nothing secret ever crossed the network. The private key stayed in a file on your laptop the entire time.
12. That is why a public key on a public web page is not a leak, and why a private key in a chat message is a catastrophe.

PLAIN42.3.2 a picture in your head#

1. Imagine a very unusual rubber stamp.
2. Only you own the stamp. It presses a pattern so intricate that nobody can carve a copy from looking at an impression.
3. You give everyone a printed picture of the pattern. That is the public key.
4. Now anyone can send you a fresh sheet of paper with today’s date and a random number on it and ask you to stamp it.
5. You stamp it and hand it back. They compare the impression with the printed picture. It matches, so it was you.
6. The random number matters more than it looks. If the paper always said the same thing, someone could keep an old stamped sheet and reuse it.
7. Because the paper is fresh every time, an old stamped sheet proves nothing about today.

Where this comparison breaks:

8. A rubber stamp makes the same mark every time. A real signature depends on both the key and the exact data, so it is different for every message.
9. And a rubber stamp can be physically stolen and used. A private key can be locked with a passphrase, so stealing the file is not always enough.
10. The honest version: some key types, RSA in particular, can also be used the other way round, to encrypt something that only the private key can open. In SSH user authentication, modern practice is signing, not encrypting.

PLAIN42.3.3 a worked example#

1. Here is a real key being made on the machine used for this chapter. Nothing is edited except line width.

$ ssh-keygen -t ed25519 -C "kedbyte@laptop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (~/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_ed25519
Your public key has been saved in ~/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:WH1E2gSLhG6FgMsha//GINoqouvX3fLxFRdMA3KMrSU
kedbyte@laptop
The key's randomart image is:
+--[ED25519 256]--+
|   ... o. .+Bo.o |
|. o   o..o Eo+o .|
| + o . .o + *  o |
|..o   oo   o    .|
|. .  .. S    . . |
| . o          o  |
|... = . ..   .   |
|+ .. = o .o .    |
|O+. .   o. .     |
+----[SHA256]-----+

2. Two files came out. id_ed25519 is private, 411 bytes. id_ed25519.pub is public, 96 bytes.
3. The fingerprint is a short hash of the public key, for humans to compare by eye. It is not a key and it is not secret.
4. The randomart is the same fingerprint drawn as a picture, because humans spot a changed picture faster than a changed string of letters.
5. Now let us prove what a private key can actually do. Sign a file:

$ echo "hello kedbyte" > msg.txt
$ ssh-keygen -Y sign -f ~/.ssh/id_ed25519 -n file msg.txt
Signing file msg.txt
Write signature to msg.txt.sig

6. Check it with only the public key:

$ ssh-keygen -Y verify -f allowed_signers -I kedbyte@laptop \
    -n file -s msg.txt.sig < msg.txt
Good "file" signature for kedbyte@laptop with ED25519 key
SHA256:WH1E2gSLhG6FgMsha//GINoqouvX3fLxFRdMA3KMrSU

7. Now change one word of the message and check the same signature again:

$ ssh-keygen -Y verify -f allowed_signers -I kedbyte@laptop \
    -n file -s msg.txt.sig < msg2.txt
Signature verification failed: incorrect signature
Could not verify signature.

8. That is the entire idea, demonstrated. The signature is bound to the exact bytes. Change one character and it stops matching.

PLAIN42.3.4 what is really happening inside#

1. Look at the public key file. It is one line with three fields separated by spaces. Here it is, wrapped for the page:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIpKlC+6wSQAkCHl
gza1Sb50MnBwsPXPD6FAeelzkUGS kedbyte@laptop

2. Field one, ssh-ed25519, is the key type. Field three, kedbyte@laptop, is a comment. It is a label for humans and has no effect on anything.
3. Field two is the key itself, encoded in base64 so it can be pasted into a text box. Decoded, it is 51 bytes with a strict internal structure.
4. That structure is a repeating pattern: a four-byte length, then that many bytes of data. Here is the real decode of the key above:

total bytes: 51
field 1: length 11 -> 'ssh-ed25519'
field 2: length 32 -> 8a4a942fbac124009021e58336b549be
                      74327070b0f5cf0fa14079e973914192

5. So the type name appears twice: once in plain text outside the base64, and once inside it. The inside copy is the one that counts.
6. The 32 bytes are the actual Ed25519 public key. That is the whole secret-free half of the pair, and it really is only 32 bytes.
7. Now the login exchange, in order.
8. Your client tells the server: I would like to authenticate as this user with this public key. Would you accept it at all?
9. The server looks the key up in that user’s authorized_keys file. If it is not there, the server says no and your client tries the next key.
10. If it is there, the server says yes, that key is acceptable. Note that nothing has been proven yet.
11. Your client now builds a block of data to sign. It contains the session identifier from the key exchange, the user name, the service name, and the public key.
12. The session identifier is the crucial part. It is unique to this one connection and could not have been known in advance.
13. Your client signs that block with the private key and sends the signature.
14. The server verifies the signature against the stored public key. If it verifies, you are authenticated.
15. Because the signed block includes the session identifier, a signature captured from an old session cannot be replayed into a new one.
16. And because the server also binds its own host key into the session identifier, a man-in-the-middle cannot forward your signature to the real server. That property is why host key checking is not optional.

TECHNICAL42.3.5 the engineer’s version#

1. Three key types are in real use, plus a dead one. Real measurements from the OpenSSH 9.6 machine used for this chapter.

2. Generation cost, averaged over five runs on the same machine:

3. RSA. Published by Rivest, Shamir and Adleman in 1978. Security rests on the difficulty of factoring a large number into two primes.
4. 1024-bit RSA is broken in practice for well-funded attackers and is refused by modern software. 2048 bits is the floor, and is what NIST Special Publication 800-57 treats as acceptable to roughly 2030.
5. 4096 bits is common because it costs almost nothing for a key you generate once, and pushes the estimated strength well past the 2030 line.
6. Beyond 4096 the returns collapse. RSA key strength grows very slowly with size while signing cost grows fast, which is why 8192 is rare.
7. The critical modern detail is the signature algorithm, not the key size. The old ssh-rsa algorithm uses SHA-1, which has been considered unsafe since the 2017 SHAttered collision by CWI Amsterdam and Google.
8. RFC 8332, published in March 2018, defines rsa-sha2-256 and rsa-sha2-512 for the same RSA keys. OpenSSH disabled ssh-rsa by default in version 8.8 in September 2021.
9. ECDSA. The elliptic curve digital signature algorithm, standardized in FIPS 186. In SSH it is used with the NIST curves P-256, P-384 and P-521.
10. ECDSA works and is widely supported, but it has two known unhappy properties. It needs a fresh random value for every signature, and a repeated or biased value leaks the private key outright.
11. That is not theoretical. The 2010 Sony PlayStation 3 signing key recovery was exactly this bug, a fixed nonce.
12. Also, some cryptographers distrust the provenance of the NIST curve constants. This is a genuine expert disagreement. One side says the constants are unexplained and should be avoided, the other says no attack has ever followed from them and the curves are fine.
13. Ed25519. EdDSA over the twisted Edwards curve Curve25519, from the 2011 paper “High-speed high-security signatures” by Bernstein, Duif, Lange, Schwabe and Yang. Curve25519 itself is Bernstein’s, from 2006.
14. It is the current recommendation for SSH, for four reasons that are concrete rather than fashionable.
15. It is deterministic: the per-signature value is derived from the key and the message, so the ECDSA nonce failure mode cannot happen.
16. It is small and fast: 32-byte public keys, 64-byte signatures, and verification that is quick on tiny hardware.
17. Its parameters were chosen by a published rule with stated rationale, so there are no unexplained constants.
18. Its implementations are naturally constant-time, which resists timing side-channel attacks without special care.
19. Ed25519 in SSH is standardized in RFC 8709, February 2020, though OpenSSH shipped it in version 6.5 in January 2014, six years earlier.
20. DSA. ssh-dss, 1024 bits, roughly 80-bit security. GitHub removed support for DSA keys entirely, permanently enforced on 15 March 2022.
21. The complete list your OpenSSH understands is one command:

$ ssh -Q key
ssh-ed25519
sk-ssh-ed25519@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
ssh-dss
ssh-rsa

22. The sk- prefixed types are hardware security key types, added in OpenSSH 8.2 in February 2020. The private key half lives on a physical device such as a YubiKey and cannot be copied off it.
23. Fingerprints default to SHA-256 in base64 since OpenSSH 6.8, March 2015. The older MD5 hex form is still available and still seen in old documentation:

$ ssh-keygen -lf id_ed25519.pub
256 SHA256:WH1E2gSLhG6FgMsha//GINoqouvX3fLxFRdMA3KMrSU
kedbyte@laptop (ED25519)

$ ssh-keygen -lf id_ed25519.pub -E md5
256 MD5:74:36:b8:87:74:1d:49:9c:09:81:62:6b:89:eb:26:ad
kedbyte@laptop (ED25519)

24. Practical recommendation, and it is not controversial: ssh-keygen -t ed25519 -C "you@machine". Use RSA 4096 only when you must talk to something old that cannot do Ed25519.

WORDS42.3.6 remember these#

Key pair — two matched files, one secret one public — an asymmetric keypair where the public half is derivable from the private half but not the reverse.

Private key — the file you never share — the secret scalar or exponent, stored under ~/.ssh/, optionally encrypted with a passphrase.

Public key — the file you paste everywhere — the shareable half, safe to publish, stored on the server in authorized_keys.

Fingerprint — a short code for comparing keys by eye — the base64 SHA-256 hash of the public key blob, printed by ssh-keygen -l.

Signature — proof that the holder of the key saw this data — the output of a signing algorithm over specified bytes, verifiable with the public key alone.

Ed25519 — the recommended modern key type — EdDSA over Curve25519, RFC 8709, 32-byte public keys, deterministic signatures.

Nonce — a number used once — a per-signature random or derived value whose reuse in ECDSA leaks the private key.

42.4 authorized_keys and how it works#

PLAIN42.4.1 in simple words#

1. On the server there is one plain text file that decides who may log in as a given user with a key.
2. It is called authorized_keys and it lives in that user’s home directory, inside a folder called .ssh.
3. Each line of the file is one public key. If your public key is on a line, you may log in as that user.
4. That really is the entire mechanism. There is no database and no clever service. It is a text file that gets read.
5. To give someone access, you append their public key as a new line. To remove access, you delete that line.
6. There is one rule that catches everybody. The file and its folder must not be writable by anyone other than the owner.
7. If they are, the server refuses to use the file at all, and says nothing useful about why.
8. The reason is sound. If any other user could edit that file, they could add their own key and become you. So SSH refuses to trust it.
9. The failure is silent from the client’s side. You get “permission denied” and no explanation.
10. GitHub has a variant of this idea for a single repository, called a deploy key. Same file format, narrower reach.

PLAIN42.4.2 a picture in your head#

1. Think of a guest list on a clipboard at the door of a private room.
2. The doorman does not know anyone’s face. He only checks the list.
3. To let someone in, you write their name on the list. To stop them, you cross it off. There is no other step.
4. Now add the rule that catches people out. The clipboard hangs on a hook in the public corridor.
5. If anyone walking past can write on it, the list is worthless. Anyone could add themselves.
6. So the doorman has a standing instruction: if the clipboard is not locked in its case, ignore the list entirely and let nobody in.
7. And he does not explain. He just says “not on the list”, because telling you about the broken case would help an attacker more than it helps you.

Where this comparison breaks:

8. A name on a list can be claimed by anyone who knows the name. A public key on a line can only be used by whoever holds the matching private key.
9. So the list being publicly readable is fine. Only the list being publicly writable is fatal.

PLAIN42.4.3 a worked example#

1. This is a real server, a real key, and a real failure, produced on the machine used for this chapter.
2. First, correct permissions. The connection works.

$ ls -l ~/.ssh/authorized_keys
-rw------- 1 root root 96 Aug 13 08:22 authorized_keys
$ ssh -i ~/.ssh/id_ed25519 -p 2222 root@127.0.0.1 'echo OK'
OK

3. Now make the file writable by everyone, which is exactly the mistake people make when they get frustrated and run chmod 666.

$ chmod 666 ~/.ssh/authorized_keys
$ ls -l ~/.ssh/authorized_keys
-rw-rw-rw- 1 root root 96 Aug 13 08:22 authorized_keys
$ ssh -i ~/.ssh/id_ed25519 -p 2222 root@127.0.0.1 'echo OK'
root@127.0.0.1: Permission denied (publickey).

4. The key did not change. The user did not change. Only the permission bits changed, and the login broke.
5. Nothing in the client message says so. It says Permission denied (publickey), which is the same message you get for a completely wrong key.
6. Now the server’s own log, from the same moment:

Connection from 127.0.0.1 port 48576 on 127.0.0.1 port 2222
Authentication refused: bad ownership or modes for file
  /root/.ssh/authorized_keys
Failed publickey for root from 127.0.0.1 port 48576 ssh2:
  ED25519 SHA256:WH1E2gSLhG6FgMsha//GINoqouvX3fLxFRdMA3KMrSU
Connection closed by authenticating user root [preauth]

7. There it is: bad ownership or modes. The truth exists, but only on the server side.
8. This is the single most useful piece of knowledge in this section. When public key login fails and you control the server, read the server log.
9. The fix is two commands:

$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/authorized_keys

PLAIN42.4.4 what is really happening inside#

1. Here is the exchange as a numbered sequence, with a diagram after it.
2. The client opens TCP to port 22 and the two sides complete the encrypted transport layer, including the server proving its host key.
3. The client sends an authentication request naming the user, the service, the method publickey, and one public key. A flag says “this is only a query”.
4. The server takes the user name and looks up that user’s home directory.
5. It checks the ownership and permission bits on the home directory, on ~/.ssh, and on ~/.ssh/authorized_keys.
6. If any of them is writable by group or by others, the server abandons the file and behaves as if no key matched. This is the silent failure.
7. Otherwise it reads the file line by line, skipping blanks and comments, looking for a line whose key blob equals the one offered.
8. If none matches, the server replies “failure” and the client tries its next key, then finally gives up with Permission denied (publickey).
9. If one matches, the server replies “this key is acceptable, now prove it”.
10. The client builds the signature block, signs it, and sends it. The server verifies with the stored public key.
11. On success the server applies any options written in front of that key on that line, then starts the session.

 client                                   server
   |                                        |
   | 1. transport set up, host key checked  |
   |--------------------------------------->|
   |                                        |
   | 2. "can I use key K as user U?"        |
   |--------------------------------------->|
   |                                        | 3. check modes on
   |                                        |    ~, ~/.ssh, file
   |                                        | 4. search file for K
   |          5. "K is acceptable"          |
   |<---------------------------------------|
   |                                        |
   | 6. signature over session id + U + K   |
   |--------------------------------------->|
   |                                        | 7. verify with K
   |          8. success + options applied  |
   |<---------------------------------------|

12. Steps 3 and 4 are where nearly all real-world failures live, and neither of them tells the client anything.

TECHNICAL42.4.5 the engineer’s version#

1. Default path is ~/.ssh/authorized_keys, set by the AuthorizedKeysFile directive in sshd_config. It accepts multiple paths and the tokens %h for home directory and %u for user name.
2. Required modes, enforced when StrictModes yes is set, which is the default:

3. The client enforces its own modes as well. A private key readable by others produces UNPROTECTED PRIVATE KEY FILE and is refused.
4. Each line is: optional comma-separated options, then key type, then base64 key, then optional comment. No line continuation exists, so a wrapped paste is the second most common cause of failure after permissions.
5. The options that matter in production:

6. restrict was added in OpenSSH 7.2, February 2016, and is the safe default because it enables all current and future restrictions, letting you re-enable individually with permit* options.
7. A real forced command, tested on the machine used for this chapter:

$ cat ~/.ssh/authorized_keys
command="echo forced: you may only run this",
no-port-forwarding,no-agent-forwarding,no-pty
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... kedbyte@laptop

$ ssh -i ~/.ssh/id_ed25519 -p 2222 root@127.0.0.1 \
    'rm -rf /important'
forced: you may only run this

8. The destructive command was sent and simply not run. The original command is still available to the forced command in the environment variable SSH_ORIGINAL_COMMAND, which is how restricted git servers work.
9. That is precisely the mechanism behind git-shell, the login shell that permits only git-upload-pack, git-receive-pack and git-upload-archive.
10. Deploy keys are GitHub’s per-repository variant. The public key is attached to one repository, not to a user account.
11. Deploy keys are read-only by default and can be granted write access when added. One key cannot be reused across repositories: GitHub rejects a duplicate.
12. GitHub lists five alternatives to deploy keys: SSH agent forwarding, HTTPS with OAuth tokens, GitHub App installation access tokens which expire after one hour, machine users, and personal access tokens.
13. On GitHub, all SSH access uses the single account name git. Your identity comes entirely from which key signed, not from the user name in the URL.
14. That is why git@github.com is the same for every person on earth, and why ssh -T git@github.com greets you by your own name.

WORDS42.4.6 remember these#

authorized_keys — the list of keys allowed in — a per-user file of public key lines consulted by sshd during publickey authentication.

StrictModes — the rule that file permissions must be tight — an sshd directive making the server refuse key files writable by group or others.

Forced command — a key that can only do one thing — the command="..." option overriding whatever the client requested.

SSH_ORIGINAL_COMMAND — what the client actually asked for — the environment variable holding the requested command when a forced command is in effect.

Deploy key — a key for one repository, not one person — a public key attached to a single GitHub repository, read-only unless granted write.

git-shell — a login shell that only allows git — a restricted shell shipped with Git permitting only the three pack transfer commands.

42.5 known_hosts and host key verification#

PLAIN42.5.1 in simple words#

1. Everyone thinks about proving themselves to the server. Almost nobody thinks about the server proving itself to them.
2. But that half exists, it runs first, and it is the half that stops somebody sitting invisibly in the middle of your connection.
3. Every SSH server has its own key pair, separate from any user’s. It is called the host key.
4. During setup, the server signs part of the conversation with its private host key. Only the real server could do that.
5. Your client checks that signature against a copy of the server’s public key it saved earlier, in a file called known_hosts.
6. And here is the awkward truth. The first time you ever connect, your client has no saved copy, so it cannot check anything.
7. So it asks you. It shows you a fingerprint and asks whether to trust it. Most people type yes without looking.
8. That approach has a name: trust on first use. You are trusting that nobody was interfering during that one first moment.
9. After that first yes, the key is saved and every later connection is properly checked.
10. If the saved key ever stops matching, SSH stops everything and prints a very loud warning. That warning is doing its job. Read it, do not delete it.

PLAIN42.5.2 a picture in your head#

1. You are meeting someone you have only spoken to by letter, and you must be sure it is really them.
2. The first meeting is the hard one. They show you a birthmark. You have no way to check it, so you write it down and decide to believe it.
3. Every meeting afterwards is easy. You look for the birthmark you wrote down. Match means it is them.
4. Now imagine you turn up and the birthmark is different.
5. Three explanations. They had it removed and told nobody. They have an identical twin who came instead. Or somebody is impersonating them.
6. You cannot tell which from looking. What you can do is stop, and go and check through a different channel before saying anything private.

Where this comparison breaks:

7. A birthmark cannot be changed at will. A host key can be legitimately replaced, and sometimes must be. So a mismatch is not automatically an attack.
8. And a person can be phoned to confirm. A server cannot. You have to find the fingerprint published somewhere trustworthy instead.

PLAIN42.5.3 a worked example#

1. All of the following is real output from the machine used for this chapter, against a small server started for the purpose.
2. The first connection, with nothing saved yet:

$ ssh -p 2222 root@127.0.0.1
The authenticity of host '[127.0.0.1]:2222' can't be
established.
ED25519 key fingerprint is
SHA256:nu2laRCiSw3wMTbG6hoH8tKukC952d+/PbROtgsD5vU.
This key is not known by any other names.
Are you sure you want to continue connecting
(yes/no/[fingerprint])?

3. Answer yes and the client writes one line into known_hosts. Here it is, truncated for page width:

$ cat ~/.ssh/known_hosts
[127.0.0.1]:2222 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA
IJJmXI2wR7keztPI78nlxu2ooGFa2+d/ke5qLMBiy5...

4. Three fields: which host, which key type, and the key. Note the host is in square brackets with the port, because it is not port 22.
5. Now the server’s host key was deliberately replaced and the same connection tried again. This is the real, complete warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@   WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now
(man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote
host is
SHA256:fwKYxD4JEmBcHpHj3WkgxSLrRh6tXIzpi/KOxz2yNrA.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of
this message.
Offending ED25519 key in ~/.ssh/known_hosts:1
  remove with:
  ssh-keygen -f '~/.ssh/known_hosts' -R '[127.0.0.1]:2222'
Host key for [127.0.0.1]:2222 has changed and you have
requested strict checking.
Host key verification failed.

6. Notice the message even tells you the exact line number and the exact command to remove it. That convenience is why people delete first and think never.
7. Notice also that the connection stopped. No password prompt appeared. SSH refused to send anything to a server it could not identify.

PLAIN42.5.4 what is really happening inside#

1. The host key check happens during the key exchange, before any authentication method is even offered.
2. The two sides compute a shared secret, then a hash covering both version strings, both algorithm lists, both key exchange values and the shared secret. That hash is the session identifier.
3. The server signs the session identifier with its private host key and sends the signature plus its public host key.
4. The client verifies the signature using the public host key it was just sent. That proves the far end holds the matching private host key.
5. But it does not prove the far end is the right machine. Anyone can generate a key pair and sign with it.
6. So the client separately compares that public host key against known_hosts. That is the step that ties the maths to a name you care about.
7. Four outcomes. Not found means prompt the user. Found and matching means proceed. Found and different means abort loudly.
8. The fourth outcome is subtler: found for a different key type. If you have an Ed25519 entry and the server offers RSA, OpenSSH says “this key is not known by any other names” rather than warning of a change.
9. When a mismatch happens, SSH refuses to continue in the default configuration, because continuing means handing your credentials to something unverified.
10. If you use password authentication and click through such a warning, you have just typed your password into whatever is in the middle. With key authentication you would have leaked a signature, which is far less valuable, but still not nothing.

TECHNICAL42.5.5 the engineer’s version#

1. known_hosts lives at ~/.ssh/known_hosts for the user and /etc/ssh/ssh_known_hosts for the system. Controlled by UserKnownHostsFile and GlobalKnownHostsFile.
2. Line format: host patterns, key type, base64 key, optional comment. Host patterns may be comma-separated, may use * and ?, and non-default ports are written [host]:port.
3. HashKnownHosts yes stores hosts as HMAC-SHA-1 hashes so a stolen file does not enumerate every server you use. Debian and Ubuntu enable this by default; upstream OpenSSH does not.
4. StrictHostKeyChecking has four values, and choosing correctly matters more than people think:

5. accept-new was added in OpenSSH 7.6, October 2017, and is the correct choice for scripts. no is dangerous because it also tolerates a changed key, which is the case you actually care about.
6. Useful commands, all real:

$ ssh-keyscan -t ed25519 -p 2222 127.0.0.1 >> known_hosts
$ ssh-keygen -F '[127.0.0.1]:2222' -f known_hosts
# Host [127.0.0.1]:2222 found: line 1
$ ssh-keygen -R '[127.0.0.1]:2222' -f known_hosts

7. A warning that must be stated plainly: ssh-keyscan asks the server what its key is. If something is impersonating the server, ssh-keyscan will faithfully record the impostor’s key.
8. So ssh-keyscan is a convenience for populating a file, not a verification tool. Verification means comparing against a fingerprint published by the operator through a different channel.
9. The GitHub 2023 rotation is the textbook legitimate change. At approximately 05:00 UTC on 24 March 2023, GitHub replaced its RSA SSH host key.
10. The cause was not a compromise of GitHub’s systems. GitHub’s RSA SSH private key had been briefly exposed in a public GitHub repository, and they rotated it as a precaution.
11. Every user with an RSA entry for github.com in known_hosts then saw the REMOTE HOST IDENTIFICATION HAS CHANGED warning. The remedy GitHub published was ssh-keygen -R github.com followed by re-adding the new key.
12. Users whose known_hosts held the Ed25519 or ECDSA entry saw nothing, because only the RSA key changed.
13. GitHub’s current published fingerprints, which you should compare against rather than trusting a prompt:

14. The full Ed25519 fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU, and the full RSA one is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
15. GitHub publishes these machine-readable at the /meta endpoint of its API, under the ssh_keys field, which is what automation should consume.
16. When is a changed host key benign? When the operator announced it, when the server was rebuilt or migrated, when you connected to a load-balanced pool with inconsistent keys, or when your DNS now resolves to a different machine because you changed networks.
17. When is it alarming? When nothing changed on your side, when it happens on an untrusted network such as public wifi, when only some connections show it, or when the operator’s published fingerprint does not match.
18. The stronger long-term answer is SSH certificates. A certificate authority signs host keys, clients trust the CA once with a @cert-authority line, and individual host key changes stop mattering. OpenSSH has supported this since version 5.4, March 2010.

WORDS42.5.6 remember these#

known_hosts — your notebook of server identities — a client-side file mapping host patterns to accepted host public keys.

Host key — the server’s identity key — the long-term key pair sshd uses to sign the session identifier during key exchange.

Trust on first use — believing the first answer you get — the TOFU model, where the initial unverified key is pinned and all later ones checked against it.

ssh-keyscan — a tool that fetches a server’s key — a scanner that records whatever key is offered, with no verification of authenticity.

Session identifier — a fingerprint of this one connection — the exchange hash from the first key exchange, bound into every authentication signature.

@cert-authority — trust a signer instead of each server — a known_hosts marker line naming a CA key that may vouch for host keys.

42.6 The SSH agent#

PLAIN42.6.1 in simple words#

1. A private key should be protected by a passphrase, so that stealing the file is not enough to steal the key.
2. But that means typing a long passphrase every single time you connect.
3. Git makes this unbearable. A fetch, a push, a status check against the server: each one is a fresh connection and a fresh prompt.
4. People solve this the worst possible way. They remove the passphrase.
5. The SSH agent is the right solution. It is a small program that runs in the background and holds your unlocked keys in memory.
6. You type the passphrase once. The agent keeps the unlocked key. Every later connection asks the agent instead of asking you.
7. Here is the important detail. The agent does not hand the key to ssh.
8. ssh sends the agent the data that needs signing. The agent signs it and sends back only the signature.
9. So the key never leaves the agent’s memory, not even to another program on your own computer.
10. The agent forgets everything when it stops, which normally means when you log out or shut down.

PLAIN42.6.2 a picture in your head#

1. Think of a notary who keeps your official seal in a locked drawer.
2. You unlock the drawer once in the morning with your combination.
3. All day, whenever you need a document sealed, you hand the document to the notary. He seals it and hands it back.
4. You never carry the seal around, and nobody else ever holds it. They only ever receive sealed documents.
5. At the end of the day the drawer locks itself and the combination is forgotten.
6. Now consider agent forwarding, which is the dangerous feature.
7. It means: when I visit another office, let that office phone my notary and ask for seals on their documents.
8. That is wonderfully convenient. It also means anyone with power over that office, while you are visiting, can phone your notary and get anything sealed.
9. They cannot steal the seal. They can use it, on whatever they like, for as long as you are connected.

Where this comparison breaks:

10. A notary would notice being asked to seal something outrageous. The agent signs opaque data and cannot tell a git push from a login to your bank’s build server.

PLAIN42.6.3 a worked example#

1. Real output, in order, from the machine used for this chapter.
2. Start an agent and look inside it. Empty.

$ eval "$(ssh-agent -s)"
Agent pid 14148
$ ssh-add -l
The agent has no identities.

3. Add a key that genuinely has a passphrase. The passphrase is typed once, here and nowhere else.

$ ssh-add ~/.ssh/id_pass
Enter passphrase for ~/.ssh/id_pass:
Identity added: ~/.ssh/id_pass (with-passphrase)
$ ssh-add -l
256 SHA256:cY2TWmZ8NY6LT0x9/BYw2D8XxqpmW4+M/C5K7i+nnOI
with-passphrase (ED25519)

4. Now connect with no -i flag naming a key, and with prompting explicitly disabled so it cannot secretly ask for anything.

$ ssh -o BatchMode=yes -p 2222 root@127.0.0.1 \
    'echo CONNECTED USING AGENT KEY'
CONNECTED USING AGENT KEY

5. That worked with no passphrase typed and no key file named. The agent supplied the signature.
6. Add a key with an automatic expiry, which is a habit worth building:

$ ssh-add -t 300 ~/.ssh/id_pass
Identity added: ~/.ssh/id_pass (with-passphrase)
Lifetime set to 300 seconds

7. And empty the agent when you walk away:

$ ssh-add -D
All identities removed.

PLAIN42.6.4 what is really happening inside#

1. The agent listens on a Unix domain socket, which is a file on disk used as a communication endpoint between programs on the same machine.
2. Its path is put in the environment variable SSH_AUTH_SOCK. That variable is how every other program finds it. There is no other discovery mechanism.
3. When ssh needs to authenticate, it connects to that socket and asks: what identities do you hold? The agent replies with public keys only.
4. ssh offers those public keys to the server. If the server accepts one, ssh builds the data to be signed and sends it to the agent.
5. The agent signs and returns the signature. ssh forwards it to the server. At no point does ssh see the private key.
6. Agent forwarding, the -A flag, changes one thing: on the remote machine, sshd creates a new socket and sets SSH_AUTH_SOCK to point at it.
7. Anything on that remote machine that can read that socket can ask your agent for signatures.
8. Crucially, the socket is a file, and the root user of that machine can read any file. So root on the remote host can use your keys.
9. Not copy them. Use them, silently, for as long as your session lasts, against every server your keys open.
10. That is why agent forwarding to a machine you do not fully control is a genuine security decision, not a convenience toggle.

  your laptop                    remote host you visited
 +-------------+               +-----------------------+
 |  ssh-agent  |               |  your shell           |
 |  holds key  |               |  SSH_AUTH_SOCK=/tmp/. |
 +------+------+               +-----------+-----------+
        | socket                           |
        |  <---- forwarded channel ------->|
        |                                  |
        |                       root here can also
        |                       talk to your agent

TECHNICAL42.6.5 the engineer’s version#

1. The agent protocol is specified in RFC 4251’s ecosystem as draft-miller-ssh-agent, published as RFC 9578 in June 2024. Before that it was a de facto standard for over twenty years.
2. Core commands and what they do:

3. ssh-add -c is underused. It makes the agent pop a confirmation dialog for every signature, which turns silent abuse over a forwarded agent into something you would notice immediately.
4. The relevant client configuration keys:

5. IdentitiesOnly yes solves a real problem. Without it, ssh offers every key the agent holds, in order. Servers commonly refuse after six failed attempts with Too many authentication failures.
6. On macOS, the system agent is integrated with the login keychain. Since macOS 10.12.2 the correct configuration is explicit:

Host *
    UseKeychain yes
    AddKeysToAgent yes
    IdentityFile ~/.ssh/id_ed25519

7. UseKeychain yes stores the passphrase in the login keychain and retrieves it automatically, so the agent is repopulated after a reboot. It is a macOS-only option and will error on Linux.
8. On Linux, gnome-keyring and KDE’s ksshaskpass provide similar integration. On Windows, OpenSSH ships an ssh-agent Windows service.
9. The safer alternative to agent forwarding is ProxyJump, added in OpenSSH 7.3, August 2016:

Host internal
    HostName 10.0.5.20
    ProxyJump bastion.example.com

10. ProxyJump tunnels a new SSH connection through the intermediate host. The intermediate host sees only encrypted bytes and never touches your agent.
11. A complete, realistic ~/.ssh/config. This file must be mode 600 and OpenSSH applies the first matching value for each key, so specific hosts must come before Host *:

# GitHub over the HTTPS port, for blocked networks
Host github.com
    HostName ssh.github.com
    Port 443
    User git
    IdentityFile ~/.ssh/id_ed25519_github
    IdentitiesOnly yes

# A work server on an odd port, through a bastion
Host build
    HostName 10.0.5.20
    Port 2222
    User deploy
    ProxyJump bastion.example.com
    IdentityFile ~/.ssh/id_ed25519_work
    IdentitiesOnly yes

# Everything else
Host *
    AddKeysToAgent yes
    ServerAliveInterval 60
    ServerAliveCountMax 3
    HashKnownHosts yes

12. You never have to guess what that file computed. ssh -G prints the final resolved settings. Real output from this chapter’s machine:

$ ssh -G demo
user root
hostname 127.0.0.1
port 2222
identitiesonly yes
identityfile /tmp/sshdemo/.ssh/id_ed25519

13. ServerAliveInterval 60 with ServerAliveCountMax 3 sends a keepalive every 60 seconds and gives up after three unanswered ones. This is the fix for connections silently dying behind a NAT device that expires idle entries.

WORDS42.6.6 remember these#

SSH agent — a background helper holding unlocked keys — a daemon exposing a signing oracle over a Unix socket, never releasing key material.

SSH_AUTH_SOCK — the address of your agent — the environment variable naming the agent’s Unix domain socket path.

Agent forwarding — letting a remote host use your agent — an SSH channel exposing your local agent socket on the remote machine, usable by its root.

ProxyJump — hop through a middle machine safely — a client directive tunnelling a fresh SSH session through an intermediate host without agent exposure.

IdentitiesOnly — only try the key I named — a directive suppressing the agent’s other keys, avoiding Too many authentication failures.

UseKeychain — let macOS remember the passphrase — an Apple-specific ssh_config option storing and retrieving passphrases from the login keychain.

42.7 Why SSH on port 22 can be blocked when 443 is not#

PLAIN42.7.1 in simple words#

1. A port is a number attached to a connection saying which service you want on that machine.
2. Web traffic over HTTPS uses port 443. SSH uses port 22.
3. Many networks do not let you reach every port on the outside world. They allow a short list and block the rest.
4. Port 443 is always on that list. If you block 443, the web stops working, and somebody complains within seconds.
5. Port 22 is often not on the list. Blocking it upsets developers, and only developers, and usually not loudly enough to matter.
6. There are also two real reasons a network operator gives for filtering 22.
7. First, port 22 is scanned constantly by automated programs looking for servers with weak passwords. Filtering it reduces that noise.
8. Second, an outbound SSH connection can carry anything at all inside it, in a form nobody can inspect. It is a perfect way to move data out unseen.
9. So the block is not aimed at you. You are collateral.
10. And the way it fails is cruel. A blocked port 22 usually does not say “no”. It says nothing at all, and your command hangs.
11. This is exactly the shape of the reader’s own fault in this book, and the fix is one of two lines of configuration.

PLAIN42.7.2 a picture in your head#

1. A building with many numbered delivery bays. Bay 443 is the main goods entrance, open all day, huge queues, everyone uses it.
2. Bay 22 is a small side entrance used by maintenance staff.
3. Management decides to weld bay 22 shut. Almost nobody notices. Bay 443 cannot be welded shut, because then the business stops.
4. Now, welding shut has two possible styles.
5. The polite style is a sign on the door: “closed, use the main entrance”. You read it, turn around, and lose ten seconds.
6. The unhelpful style is no sign at all. You stand there. You knock. You wait. You knock again. Eventually you give up, knowing nothing.
7. Networks overwhelmingly choose the second style, on purpose, because a sign tells a stranger which doors exist.

Where this comparison breaks:

8. A welded door is permanent and obvious on inspection. A network filter can be selective by time, by source address or by destination, so the same command can work for your colleague and hang for you.

PLAIN42.7.3 a worked example#

1. This is real, measured on the machine used to write this chapter, which happens to sit behind exactly this kind of filtering.
2. Port 22 to GitHub, with a generous timeout:

$ time ssh -o ConnectTimeout=30 -T git@github.com
ssh: connect to host github.com port 22: Connection timed out

real  0m30.057s

3. Thirty seconds, and then a timeout. Nothing came back at any point.
4. Port 443 to the same host name, in the same shell, one second later:

$ time bash -c 'exec 3<>/dev/tcp/github.com/443 && echo OPEN'
OPEN

real  0m0.021s

5. Twenty-one milliseconds. The network is fine. DNS is fine. GitHub is fine. Only port 22 is unreachable.
6. Now compare that with a port where something actively refuses. Nothing is listening on port 2200 on the local machine:

$ time ssh -o ConnectTimeout=10 -p 2200 root@127.0.0.1
ssh: connect to host 127.0.0.1 port 2200: Connection refused

real  0m0.015s

7. Fifteen milliseconds and a clear refusal. That is what a healthy “no” looks like.
8. Put the three side by side and the diagnosis writes itself.

9. The timing is the tell. Milliseconds means somebody answered. Tens of seconds means silence.

PLAIN42.7.4 what is really happening inside#

1. Chapter 34 of this book covered two ways to block a packet, and the difference decides your whole afternoon. Here it is again in one line each.
2. REJECT means refuse and say so. The filter sends back a TCP reset or an ICMP error. Your program fails instantly with “connection refused”.
3. DROP means refuse and say nothing. The packet is deleted. No reply is generated at all.
4. With DROP, your computer does not know anything happened. It assumes the packet was lost, so it does what TCP always does: it sends it again.
5. The retries are spaced out with doubling gaps. One second, two, four, eight, sixteen. Then it gives up and reports a timeout.
6. That is why a blocked port 22 hangs for tens of seconds rather than failing instantly. Every second of that wait is your machine politely retrying.
7. Firewall administrators choose DROP deliberately. A REJECT confirms that a host exists at that address, which helps someone mapping your network.
8. So silence is a security choice that costs you diagnosis time.
9. Applying it to the reader’s own session: curl -v https://github.com printed Trying 20.207.73.82:443... and then produced nothing for fifteen seconds. That is the same signature at a different port.
10. What that proves is narrow and must be stated carefully. It proves no response arrived. It is consistent with a silent drop somewhere on the path. It does not identify who dropped it, and it does not prove the server was down.
11. What made the reader’s case conclusive was the same request succeeding over mobile data seconds later. Same destination, different path, different result.

TECHNICAL42.7.5 the engineer’s version#

1. The exact test to determine whether port 22 is blocked, in order, and each step distinguishes something specific.

# 1. Does the name resolve? (rules out DNS)
$ dig +short github.com
20.207.73.82

# 2. Can TCP reach port 22 at all? (the real question)
$ nc -vz -w 5 github.com 22

# 3. Same for 443, as a control
$ nc -vz -w 5 github.com 443

# 4. If nc is absent, bash can do it
$ time bash -c 'exec 3<>/dev/tcp/github.com/22'

# 5. GitHub's own SSH check, most direct of all
$ ssh -vT -o ConnectTimeout=10 git@github.com

2. Read the results this way:

3. Fix one: switch the remote to HTTPS. One command per repository, or a global rewrite for all of them.

$ git remote set-url origin \
    https://github.com/owner/repo.git

# or globally, for every GitHub remote at once
$ git config --global \
    url."https://github.com/".insteadOf "git@github.com:"

4. Fix two: use GitHub’s SSH over the HTTPS port. GitHub runs an SSH endpoint on port 443 at a different host name.
5. The host name is ssh.github.com, not github.com. This matters. GitHub’s documentation states it explicitly because people get it wrong.
6. Test it first:

$ ssh -T -p 443 git@ssh.github.com
Hi USERNAME! You've successfully authenticated, but GitHub
does not provide shell access.

7. Then make it permanent with the exact block GitHub publishes, in ~/.ssh/config:

Host github.com
    Hostname ssh.github.com
    Port 443
    User git

8. After that block, your existing git@github.com:owner/repo.git remotes need no change at all. ssh rewrites the destination before connecting.
9. This is real SSH, not SSH wrapped in TLS. It is the SSH protocol speaking on TCP port 443. A firewall filtering by port number is satisfied.
10. Therefore it does not help against a proxy that inspects protocols, only against a port-number filter. GitHub’s own documentation notes that proxy servers may interfere.
11. Comparison of the two fixes, since they are not equivalent:

12. A third option exists and is worth knowing: ProxyCommand with a tool such as corkscrew or nc tunnels SSH through an HTTP proxy using the CONNECT method. It is fiddly and usually not needed if fix one works.
13. On the honest side: none of this is a way around a policy that forbids the traffic. Using SSH on 443 to evade a corporate control is a conversation with your employer, not a technical question.

WORDS42.7.6 remember these#

Port — a number naming a service on a machine — a 16-bit TCP or UDP endpoint identifier; 22 for SSH, 443 for HTTPS by IANA assignment.

DROP — block and say nothing — a filter action discarding the packet with no reply, producing a client-side timeout after retries.

REJECT — block and say so — a filter action returning a TCP RST or ICMP unreachable, producing an immediate Connection refused.

Egress filtering — controlling what may leave a network — outbound firewall policy, commonly permitting only 80, 443 and DNS.

ssh.github.com — GitHub’s SSH server on the web port — the host name serving the SSH protocol on TCP 443 for networks blocking 22.

ProxyCommand — a custom way to reach a host — an ssh_config directive running an external program to establish the transport.

42.8 HTTPS authentication for git#

PLAIN42.8.1 in simple words#

1. When git talks over HTTPS, it makes ordinary web requests, and it proves who you are the way web requests always have: a user name and a secret.
2. For many years that secret was your actual GitHub account password.
3. That stopped. On 13 August 2021, GitHub stopped accepting account passwords for git operations entirely.
4. The reason is straightforward. Your account password unlocks everything you own, cannot be limited, cannot be given to one script, and cannot be cancelled without changing it everywhere.
5. In its place you use a token, which is a long random string that acts like a password but has rules attached to it.
6. A token can be limited to certain actions. It can expire. It can be cancelled on its own without touching anything else.
7. Now the practical problem. You cannot type a forty-character random string before every push.
8. So git has credential helpers. A credential helper is a small program that stores your token and hands it back when git asks.
9. Where it stores it depends on the helper, and the differences matter. Some put it in your operating system’s encrypted vault. One writes it to a plain text file.
10. Knowing which helper you use, and being able to look inside it and clear it, is a basic skill.

PLAIN42.8.2 a picture in your head#

1. Think of a building where the receptionist checks a pass every time you go through a door.
2. Carrying the pass in your hand and showing it forty times a day is tiring.
3. So you give it to a locker at the entrance. Whenever you approach a door, the locker sends the pass details for you.
4. There are different lockers. A steel one with a combination that only opens when you are logged in. And a cardboard box on the counter with your pass sitting in it, visible to anyone who looks.
5. Both work. Both save you the same effort. Only one is a good idea.
6. And when you lose your pass, or you suspect somebody copied it, you need to do two separate things.
7. Cancel the pass at the security office, so the old one stops working. And empty the locker, so your own building stops trying to use the dead one.

Where this comparison breaks:

8. A physical pass is one object. A token can be copied silently and used from anywhere in the world at the same time as you are using it, with no sign at the door.

PLAIN42.8.3 a worked example#

1. Real output from the machine used for this chapter, using git’s own credential commands. These are the exact operations a helper performs.
2. Choose a helper. Here the insecure store helper is chosen deliberately, so we can look at what it writes:

$ git config --global credential.helper \
    'store --file=/tmp/demo/.git-credentials'

3. Save a credential. This is what happens automatically the first time you push and type a token:

$ printf 'protocol=https\nhost=github.com\nusername=reader
password=ghp_EXAMPLEexampleEXAMPLEexample1234\n\n' \
  | git credential approve

4. Now look at what landed on disk:

$ ls -l /tmp/demo/.git-credentials
-rw------- 1 root root 63 Aug 13 08:24 .git-credentials
$ cat /tmp/demo/.git-credentials
https://reader:ghp_EXAMPLEexampleEXAMPLE...@github.com

5. That is your token, in a plain readable file. The permissions are 600, owner only, but there is no encryption at all.
6. Ask git what it would send for a given host. This is the command to run when you suspect a stale credential:

$ printf 'protocol=https\nhost=github.com\n\n' \
  | git credential fill
protocol=https
host=github.com
username=reader
password=ghp_EXAMPLEexampleEXAMPLEexample1234

7. And clear it, which is what you do the moment a token is rotated or leaked:

$ printf 'protocol=https\nhost=github.com\nusername=reader
password=ghp_EXAMPLEexampleEXAMPLEexample1234\n\n' \
  | git credential reject
$ wc -c < /tmp/demo/.git-credentials
0

8. The file is now empty. Note carefully: this cleared the local copy. It did nothing whatsoever to the token on GitHub’s side.

PLAIN42.8.4 what is really happening inside#

1. Git never asks a helper for a credential until it needs one.
2. It first makes the request without any credential. If the server is happy, nothing else happens. This is why cloning a public repository never prompts.
3. If the server responds with HTTP status 401 and a WWW-Authenticate header, git knows a credential is required.
4. Git then runs the configured helper with the subcommand get, feeding it the protocol, host and path on standard input.
5. The helper replies with username= and password= lines, or says nothing if it has none stored.
6. If no helper has one, git prompts you on the terminal.
7. Git then retries the request with an Authorization: Basic header. That header is the user name and token joined with a colon and base64-encoded.
8. Base64 is an encoding, not encryption. It hides nothing. What protects the token is that the whole request travels inside TLS.
9. If the retried request succeeds, git runs the helper again with store, so a working credential is remembered.
10. If it fails with 401, git runs the helper with erase, so a wrong credential is not offered forever.
11. That last step is why a bad cached token often self-corrects, and why a token that is valid but lacks a permission does not: that is a 403, not a 401, so git does not erase it.

 git ---> GET /repo.git/info/refs        ---> server
 git <--- 401 + WWW-Authenticate         <--- server
 git ---> helper "get" (host, protocol)
 git <--- helper "username=..., password=..."
 git ---> GET again + Authorization: Basic ...
 git <--- 200 OK
 git ---> helper "store"  (remember this one)

TECHNICAL42.8.5 the engineer’s version#

1. The credential helper protocol is a documented text protocol. A helper is any executable named git-credential-NAME, invoked with one of get, store or erase.
2. The helpers you will actually meet:

3. store is documented by git itself as insecure. Its own manual page says the credentials are stored unencrypted. Use it only on a machine where the whole disk is encrypted and single-user, and preferably not then.
4. cache keeps credentials in memory in a background daemon with a default timeout of 900 seconds, adjustable with --timeout.
5. Git Credential Manager, formerly Git Credential Manager Core, is the cross-platform helper maintained by GitHub. It handles OAuth device flow, so you get a browser login instead of pasting a token.
6. Configuration is layered, and this catches people out:

$ git config --show-origin --get-all credential.helper
file:/etc/gitconfig     manager
file:~/.gitconfig       osxkeychain

7. Multiple helpers are tried in order until one returns a credential. An empty value, credential.helper=, resets the list, which is the way to disable an inherited system-level helper.
8. Per-host configuration is possible and is the right way to keep work and personal accounts separate:

[credential "https://github.com"]
    username = work-account
    helper = osxkeychain

9. Inspecting and clearing on macOS, which is the reader’s platform:

# see it
$ printf "protocol=https\nhost=github.com\n\n" \
  | git credential-osxkeychain get

# delete it
$ printf "protocol=https\nhost=github.com\n\n" \
  | git credential-osxkeychain erase

10. The graphical route on macOS is the Keychain Access application, searching for github.com and deleting the internet password entry.
11. On Windows: cmdkey /list in a command prompt, or Credential Manager in Control Panel, under Windows Credentials.
12. On Linux with libsecret: secret-tool search server github.com.
13. The HTTP conversation is worth seeing once. Real capture from this chapter’s machine against a public repository, so no credential was needed:

GET /octocat/Hello-World.git/info/refs?service=git-upload-pack
Git-Protocol: version=2
<= HTTP/1.1 200 OK
<= Content-Type: application/x-git-upload-pack-advertisement

14. Against a repository the requester cannot see, unauthenticated, the same endpoint returns 401 rather than 404. Real result:

$ curl -s -o /dev/null -w "status=%{http_code}\n" \
  "https://github.com/octocat/not-a-real-repo.git/info/refs\
?service=git-upload-pack"
status=401

15. That 401 for a repository that does not exist is deliberate. GitHub will not confirm or deny the existence of a private repository to an anonymous requester. Remember this when we reach Repository not found.
16. The user name in an HTTPS git URL is almost always ignored by GitHub when a token is supplied. x-access-token, your login, or anything else usually works. The token identifies you.
17. Never put a token in the remote URL itself. It lands in .git/config in plain text, in your shell history, and in the output of git remote -v, which people paste into bug reports.

WORDS42.8.6 remember these#

Credential helper — a program that remembers your token — an executable implementing git’s get/store/erase credential protocol.

Basic authentication — user name and secret sent in a header — HTTP authentication, RFC 7617, sending base64 of user:secret, safe only inside TLS.

401 versus 403 — “who are you” versus “not allowed” — 401 means no or invalid credential and triggers a prompt; 403 means the credential is valid but insufficient.

store helper — the one that writes plain text — a helper persisting credentials unencrypted in ~/.git-credentials.

Keychain — the operating system’s encrypted vault — macOS Keychain, Windows Credential Manager or a libsecret provider, unlocked with your login.

git credential fill — ask git what it would send — a plumbing command running the helper chain and printing the resolved credential.

42.9 Personal access tokens#

PLAIN42.9.1 in simple words#

1. A token is a long random string that stands in for you when a program talks to a service.
2. It is not derived from your password. Cancelling it does not affect your password, and changing your password does not necessarily cancel it.
3. That independence is the point. A token is a separate identity you can hand to one script, on one machine, for one purpose.
4. A token has three properties a password does not have, and each one matters.
5. It can be scoped, meaning limited to a specific list of things it is allowed to do.
6. It can expire, meaning it stops working on a date you chose, whether or not you remember it exists.
7. It can be revoked individually, meaning you cancel this one without disturbing the other nine tokens on your other nine machines.
8. GitHub offers two kinds. The older classic tokens, which use a list of broad permission names. And fine-grained tokens, which pick individual repositories and individual permissions.
9. A leaked token is an emergency, not an inconvenience. Anyone holding it is you, for everything that token can reach, immediately.
10. There is no clever recovery. You revoke it, you create a new one, you replace it everywhere, and you check what was done while it was out.

PLAIN42.9.2 a picture in your head#

1. Think of a hotel with two kinds of key.
2. The master key opens every room, the safe, the office and the roof. That is your account password.
3. The other is a plastic card cut for room 412 only, valid until Friday, and the front desk can cancel it in one click without recutting anyone else’s.
4. If a cleaner needs into room 412 on Tuesday, you give them the card, not the master key.
5. If the card is lost, you cancel it. Nothing else changes. Nobody else is inconvenienced.
6. If the master key is lost, the entire hotel has to be rekeyed.
7. Now add the part that has no hotel equivalent. A copied card is invisible. The lock cannot tell the difference and neither can you.
8. So the card also has a printed expiry date, so that a copy you never learned about dies on its own.

Where this comparison breaks:

9. A hotel card must be physically present at the door. A token can be used from another continent, by a program, thousands of times a minute.
10. And a hotel lock logs the card number. Token use is logged too, but you have to go and look, and almost nobody does until something breaks.

PLAIN42.9.3 a worked example#

1. GitHub tokens begin with a short prefix that tells you what kind they are. This is a deliberate design so that scanners can find them.

2. Using one at the command line, once, without saving it anywhere:

$ git clone https://TOKEN@github.com/owner/repo.git

3. That works, and it is a bad habit, because the token is now in the shell history and in .git/config.
4. The better single-use form pipes it in and never stores it:

$ echo "TOKEN" | gh auth login --with-token

5. Rotating a token is a fixed procedure. Do these in order, because the order prevents an outage:
6. One: create the new token with the same scopes, and note the expiry.
7. Two: update every place that uses it. Your keychain, your CI secrets, the .netrc on the build box, the deployment config.
8. Three: verify the new one works, with an actual operation, not a guess.
9. Four: only now, revoke the old one.
10. Five: watch for failures for a day, because there is always one forgotten machine.
11. If instead the token has leaked, that order inverts. Revoke first, immediately, and accept the outage. A live leaked token is worse than downtime.

PLAIN42.9.4 what is really happening inside#

1. A token is a record in GitHub’s database. It has an owner, a creation time, an expiry time, a list of permissions and a last-used timestamp.
2. What you are shown once, at creation, is the token string. GitHub does not store that string in a readable form; it stores a hash of it.
3. That is why the page says you will never see it again. They genuinely cannot show it to you.
4. When you use it, the server hashes what you sent and looks up the record. If the record exists, is not expired and is not revoked, you are authenticated.
5. Then, and only then, does it check whether the permissions on that record allow the specific operation you asked for. Those are two separate steps.
6. Keeping them separate explains the whole of section 42.10. Being authenticated does not mean being allowed.
7. Secret scanning is GitHub scanning content pushed to it, looking for strings that match known credential patterns.
8. This is why the prefixes exist. ghp_ followed by the right number of base62 characters with a checksum is machine-recognisable with almost no false positives.
9. GitHub runs the same scanning for over five hundred partner patterns: cloud provider keys, payment provider keys, and so on.
10. When GitHub finds its own token in a public repository, it revokes it automatically. When it finds a partner’s, it notifies the partner, who typically revokes it too.
11. Push protection is the same detection moved earlier. Instead of finding the secret after it is pushed, the push itself is refused.
12. That distinction is enormous. A secret that reached the server must be treated as leaked forever, even after you rewrite history, because the object may have been fetched or cached.
13. A secret blocked by push protection never arrived. Nothing leaked. You fix your commit and move on.
14. Push protection has been enabled by default for public repositories since
    2024. For private repositories it is part of GitHub’s paid secret protection product.

TECHNICAL42.9.5 the engineer’s version#

1. Classic versus fine-grained, precisely:

2. GitHub’s own documentation states the case bluntly: a classic token “will grant access to all repositories within the organizations that you have access to, as well as all personal repositories”.
3. That sentence is the whole argument for fine-grained tokens. A classic token on a laptop is a key to everything that laptop’s owner can reach.
4. Fine-grained tokens are limited to resources owned by a single user or organization, can be limited to specific repositories, and are granted specific permissions rather than broad scopes.
5. Organization owners can require approval before a fine-grained token may act on organization resources, which is a control classic tokens never had.
6. Expiry: GitHub supports preset periods and a custom date. Fine-grained tokens gained optional non-expiring lifetimes and rotation policies in a change announced on 18 October 2024, and enterprises can enforce maximum lifetimes.
7. GitHub automatically removes personal access tokens that have not been used for a year. That is a useful backstop, not a strategy.
8. Practical expiry guidance, and this is a convention rather than a rule:

9. Token format, since GitHub’s April 2021 redesign: a fixed prefix, an underscore, then base62 characters, ending in a checksum. The checksum lets a scanner reject a random-looking string cheaply before doing any lookup.
10. Auditing what a token can do, without guessing. The response headers of an authenticated API call name the scopes:

$ curl -sI -H "Authorization: Bearer $TOKEN" \
    https://api.github.com/user | grep -i x-oauth
x-oauth-scopes: repo, read:org
x-accepted-oauth-scopes: 

11. x-oauth-scopes is what your token has. x-accepted-oauth-scopes is what that particular endpoint would accept. Comparing the two diagnoses most permission failures in one command.
12. With the GitHub CLI the equivalent is gh auth status, which prints the scopes of the stored token.
13. Leak response, in order, and treat this as a checklist:
14. Revoke the token in GitHub settings. This is instant and unconditional.
15. Rotate anything the token could have reached: deploy keys, downstream secrets, database passwords stored in repositories it could read.
16. Review the account’s security log and, for organizations, the audit log, filtering by the period the token was live.
17. Check for new SSH keys, new deploy keys, new webhooks, new collaborators and changed workflow files. Those are the standard persistence mechanisms.
18. Rewriting git history does not undo a leak. Assume the secret is public from the moment it was pushed.
19. Never commit a token, even to a private repository. Private repositories get forked, made public, and cloned onto laptops that later get lost.

WORDS42.9.6 remember these#

Personal access token (PAT) — a password substitute you can limit and cancel — a server-side credential record with scopes, expiry and independent revocation.

Classic token — the older broad kind — a PAT carrying OAuth-style scopes across every repository and organization the user can reach.

Fine-grained token — the newer narrow kind — a PAT limited to one owner, a chosen repository list and individual permissions.

Revocation — cancelling one credential — deleting the server-side token record, taking effect immediately and independently of other credentials.

Secret scanning — the service that hunts for leaked credentials — pattern matching over pushed content, with automatic revocation for GitHub’s own tokens.

Push protection — refusing the push instead of cleaning up after it — pre-receive secret detection that blocks the write, so nothing ever leaks.

42.10 Scopes, properly#

PLAIN42.10.1 in simple words#

1. A scope is a named permission attached to a credential.
2. The name is a short string, like repo or workflow. It sits in the token’s record on the server, not in anything you hold.
3. The server checks it at the exact moment you try to do something. Not when you log in. At the moment of the action.
4. That timing is the thing to understand. You can be perfectly, provably authenticated and still be refused.
5. Authentication answers “who are you”. Authorization answers “may you do this”. They are separate questions asked at different times.
6. So a token with no workflow scope will happily let you clone, fetch, and push ordinary changes. It will refuse one specific kind of push.
7. The reader hit exactly this. A push that changed a file under .github/workflows/ was refused, while everything else worked.
8. The reason is not arbitrary. A file under .github/workflows/ is not ordinary content. It is instructions for GitHub’s own machines.
9. Those machines run with access to the repository’s secrets. So changing that file is changing what code runs with those secrets.
10. GitHub therefore treats it as a separate, higher permission that you must ask for on purpose.

PLAIN42.10.2 a picture in your head#

1. Imagine an office pass that opens the front door, the stairs and your own floor.
2. You walk in, you go up, you work. Everything is fine. You are definitely a recognised employee.
3. Then you try the server room. The reader flashes red. Not because your pass is fake, but because your pass does not include that room.
4. Nobody has questioned your identity. They know exactly who you are. The answer is still no.
5. Now the important part: why is the server room separate at all?
6. Because inside it you could reconfigure the machines that everyone else’s work runs on, and give yourself access to everything they touch.
7. That is precisely the relationship between a normal file and a CI workflow file. One is a document. The other is a change to the factory.

Where this comparison breaks:

8. A door either opens or it does not, and it decides in one instant. A scope check can depend on the content of what you are pushing, which path it touches, and even whether an identical file already exists on another branch.

PLAIN42.10.3 a worked example#

1. The scopes that actually matter in daily work. These are the exact names GitHub uses.

2. GitHub’s own words for the two that cause the most confusion.
3. repo “grants full access to public and private repositories including read and write access to code, commit statuses, repository invitations, collaborators, deployment statuses, and repository webhooks”.
4. workflow “grants the ability to add and update GitHub Actions workflow files”. And then a detail almost nobody knows, quoted exactly.
5. “Workflow files can be committed without this scope if the same file (with both the same path and contents) exists on another branch in the same repository.”
6. That exception explains a confusing symptom. Merging a branch that already contains the workflow file can succeed while the original push failed.
7. Note that repo does not include workflow. It is not a hierarchy. The broadest normal scope still stops at the workflow directory.
8. read:org is the one people forget. Without it, a token cannot list the organizations you belong to, so tools report you as having no organizations at all.

PLAIN42.10.4 what is really happening inside#

1. Here is the reader’s exact failure, and the message is worth reading character by character.

$ git push origin main
Enumerating objects: 7, done.
Writing objects: 100% (4/4), 412 bytes, done.
To https://github.com/owner/repo.git
 ! [remote rejected] main -> main (refusing to allow a
   Personal Access Token to create or update workflow
   `.github/workflows/ci.yml` without `workflow` scope)
error: failed to push some refs to
'https://github.com/owner/repo.git'

2. Read what already succeeded before the refusal, because it tells you where the failure was not.
3. Enumerating objects and Writing objects both completed. The network worked. TLS worked. Authentication worked.
4. Your commits were transmitted to GitHub. All of them. The bytes arrived.
5. The refusal came from git-receive-pack on GitHub’s side, after receiving everything, when it inspected what was about to be applied.
6. Then it declined to update the branch reference. Nothing was written. The objects were transmitted and discarded.
7. That is what ! [remote rejected] means, and it is why this is not a transport failure. A transport failure cannot produce a sentence about scopes, because a broken transport cannot produce sentences at all.
8. The phrasing refusing to allow is also deliberate. GitHub is not saying it failed. It is saying it decided.
9. Here is the same shape reproduced locally on the machine used for this chapter, with a server-side hook standing in for GitHub’s own check. Real output:

$ git push origin main
remote: error: refusing to allow a Personal Access Token
remote: to create or update workflow
remote: `.github/workflows/ci.yml` without `workflow` scope
To /tmp/gitgate/srv/repo.git
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to '/tmp/gitgate/srv/repo.git'

10. And the push immediately before it, which touched only README.md, succeeded:

$ git push origin main
To /tmp/gitgate/srv/repo.git
 * [new branch]      main -> main

11. Same server, same credential, same network, same second. The only variable was which file the commit touched.
12. That comparison is the proof that this is a content-dependent authorization decision, taken on the server.

TECHNICAL42.10.5 the engineer’s version#

1. The fix, in order, and it is short:
2. Open GitHub, Settings, Developer settings, Personal access tokens, Tokens (classic). Select the token in use.
3. Tick workflow. Save. The scope is added to the existing token; you do not have to create a new one for classic tokens.
4. If the token was already cached, git will keep using the same string, and it now has the scope. No local change is needed.
5. If you create a new token instead, clear the cached one first, or git will keep sending the old one:

$ printf "protocol=https\nhost=github.com\n\n" \
  | git credential reject

6. With the GitHub CLI, re-authorize with the extra scope in one command:

$ gh auth refresh -h github.com -s workflow

7. For fine-grained tokens, the equivalent is the repository permission “Workflows: Read and write”.
8. For a GitHub App, it is the workflows repository permission, and the app must be installed on the repository.
9. Verify before retrying, rather than pushing and hoping:

$ curl -sI -H "Authorization: Bearer $TOKEN" \
    https://api.github.com/user | grep -i x-oauth-scopes
x-oauth-scopes: repo, workflow

10. Now generalise, because all three failures print “push failed”. This table is the most useful thing in the chapter.

11. Distinguish them by four observations, in this order.
12. Did objects transfer? Writing objects: 100% means the network and the credential both worked. Everything after that is a policy decision.
13. Was a credential requested? A prompt or a 401 means authentication failed, not authorization.
14. Is there a reason in parentheses after ! [remote rejected]? A sentence means a server rule. A bare [rejected] with non-fast-forward means your branch is behind and no permission is involved.
15. How long did it take? Instant means somebody answered. Fifteen to thirty seconds of silence means the network.
16. A worked triage, applying the table to the four cases:

Case A: 30 s, "Connection timed out"
  -> network. Port filtered. See 42.7.

Case B: "remote: Invalid username or password"
  -> authentication. Token wrong, expired or revoked.

Case C: "! [remote rejected] ... without `workflow` scope"
  -> authorization. Credential fine. Add the scope.

Case D: "! [rejected] ... (non-fast-forward)"
  -> neither. Fetch and rebase or merge.

17. One further separation the reader met in the same session. Plain git kept working while the GitHub API did not.
18. That is the same split at a larger scale: git the tool, running locally against local objects, versus GitHub the service, reachable only over the network and enforcing its own rules.
19. The honest version: the workflow scope is not a complete defence. Someone with repo scope can still change a file that a workflow executes, such as a build script the workflow calls. The scope protects the workflow definition, not everything the workflow does.
20. That is why serious repositories also use branch protection, required reviews on the .github/ path with a CODEOWNERS file, and environment protection rules on the jobs that hold real secrets.

WORDS42.10.6 remember these#

Scope — a named permission on a credential — a string in the token record that the server checks per operation, such as repo or workflow.

Authentication — establishing who you are — verifying a credential against a stored record; failure produces 401.

Authorization — establishing what you may do — checking permissions for a specific operation; failure produces 403 or a rejection message.

! [remote rejected] — the server chose not to apply this — a per-ref failure from git-receive-pack, always accompanied by a reason.

Non-fast-forward — your branch is behind, not forbidden — a rejection because the update would discard commits, unrelated to any permission.

x-oauth-scopes — the header that lists what your token has — a GitHub API response header naming the token’s granted scopes.

42.11 OAuth, tokens, SSH keys and GitHub Apps#

PLAIN42.11.1 in simple words#

1. There are four different ways a program or a person can be identified to GitHub, and people mix them up constantly.
2. An SSH key identifies a machine you sit at. It lives in a file on that machine and never moves.
3. A personal access token identifies you as a person, to a program you gave it to. It is a string, so it can be copied.
4. An OAuth authorization is what happens when you click “sign in with GitHub” on someone else’s website. You grant that site permission to act as you.
5. A GitHub App is a separate identity of its own. It is not you. It is installed on specific repositories and acts in its own name.
6. The difference that matters most is blast radius: if this credential leaks, how much can the finder reach, and for how long.
7. An SSH key with no passphrase, sitting on a stolen laptop, reaches every repository you can reach, forever, until you notice.
8. A classic token reaches the same, from anywhere in the world, with no laptop required.
9. A GitHub App installation token reaches only the repositories the app is installed on, only with the permissions granted, and it dies after one hour.
10. For automated systems there is a fifth and better option that involves no stored secret at all. We come to it at the end of this section.

PLAIN42.11.2 a picture in your head#

1. Four ways to get a package delivered into a building.
2. The SSH key is a key cut for your own front door. It works only when you are standing there. Losing it is bad but local.
3. The personal token is your signature on a delivery authorization. Anyone with a photocopy can use it anywhere, until you cancel it.
4. The OAuth grant is you signing a standing instruction that a courier company may collect on your behalf. You can withdraw it, but while it stands, they act as you.
5. The GitHub App is a contractor with their own uniform, their own badge, and a written list of which rooms they may enter. When they do something, the log says the contractor did it, not you.
6. That last property is underrated. When a person leaves the company, their key, token and OAuth grants all die with their account. The contractor keeps working.

Where this comparison breaks:

7. A contractor can be phoned and asked what they are doing. An app acts in milliseconds, thousands of times, and the only record is the audit log.

PLAIN42.11.3 a worked example#

1. The four identities compared on the properties that decide which to use.

2. And the blast radius if each one leaks:

3. Choosing, as a short decision list:
4. A human on their own laptop, working daily: an SSH key with a passphrase, held in the agent.
5. A human running a script occasionally: a fine-grained token, short expiry, limited to the repositories the script touches.
6. A server that must pull one repository: a deploy key, read-only, on that server only.
7. A product integrating with GitHub for many customers: a GitHub App.
8. A CI pipeline talking to a cloud provider: neither of these. Use OIDC, which is the next block.

PLAIN42.11.4 what is really happening inside#

1. The problem with a long-lived credential in a CI system is that it must be stored somewhere the CI system can read.
2. Which means anyone who can make the CI system print things, or run arbitrary code, can read it. And CI systems exist to run code that people wrote today.
3. The modern answer removes the stored secret entirely. It is called OIDC federation, and the idea is simple even if the acronyms are not.
4. When a job starts, GitHub creates a short-lived signed statement describing that exact job. Which repository, which branch, which workflow, which run.
5. That statement is signed by GitHub, and anyone can check the signature against GitHub’s published verification keys.
6. The job hands that statement to the cloud provider and says: I am this workflow in this repository, please give me credentials.
7. The cloud provider has been told in advance which statements to accept. For example, only jobs from owner/repo on the main branch.
8. It verifies the signature, checks the description matches its rule, and issues its own short-lived credential, valid for perhaps an hour.
9. Nothing long-lived was ever stored. There is no secret in the repository, no secret in the CI settings, and nothing to rotate.
10. If someone steals the statement, it expires in minutes and only works for that one narrow rule.

 GitHub Actions job
   |  1. request token for audience "aws"
   v
 GitHub OIDC issuer
   |  2. signed JWT: repo, ref, workflow, run_id
   v
 the job
   |  3. present JWT to the cloud provider
   v
 cloud provider
   |  4. verify signature, match trust policy
   |  5. issue short-lived credentials
   v
 the job now has 1-hour credentials, no stored secret

TECHNICAL42.11.5 the engineer’s version#

1. SSH keys authenticate a client to a server using public key cryptography. On GitHub they are attached to a user account or, as deploy keys, to a repository. They carry no scopes: an account SSH key can do whatever that account can do over git.
2. That is a genuine limitation. There is no such thing as a read-only account SSH key on GitHub. If you need read-only, you need a deploy key or a fine-grained token.
3. Personal access tokens are bearer credentials. Possession is authorization. They carry scopes and expiry as covered in 42.9 and 42.10.
4. OAuth is the delegation framework in RFC 6749, October 2012, with the current best-practice profile in RFC 9700, January 2025. Three parties: you the resource owner, the app, and GitHub the authorization server.
5. The user-facing difference from a token is that you never hand the app your credential. You are redirected to GitHub, you approve a named list of scopes, and the app receives a token bound to that grant.
6. OAuth App tokens act entirely as you. Every action appears in the audit log as your action. Revoking the grant kills the token.
7. GitHub Apps are first-class actors. Key properties:

8. A GitHub App authenticates in two stages. It signs a JSON Web Token with its own private key to prove it is the app, then exchanges that for an installation access token scoped to one installation.
9. That installation token expires after one hour, which is stated in GitHub’s own documentation as one of the reasons to prefer apps over deploy keys.
10. Apps also survive staff turnover. A token or key tied to a departing employee dies when the account is deprovisioned, usually at three in the morning, in the middle of a deployment.
11. OIDC federation for CI. OpenID Connect is an identity layer on top of OAuth 2.0. GitHub Actions runs an OIDC provider that issues a JSON Web Token per job.
12. Claims in that token include sub, which encodes the workflow context such as repo:octo-org/octo-repo:environment:prod, plus aud, repository, actor, ref, workflow and run_id.
13. The cloud side is configured with a trust policy matching those claims. A correct policy pins at least the repository and usually the branch or environment.
14. GitHub states three benefits: no cloud secrets duplicated as long-lived GitHub secrets, granular control through the provider’s own access tools, and automatic rotation because each token expires when the job ends.
15. A minimal workflow needs one permission block and one action:

permissions:
  id-token: write
  contents: read

16. Without id-token: write the job cannot request an OIDC token at all, and the failure message is unhelpful, so this is worth memorizing.
17. GITHUB_TOKEN, the automatic token inside every Actions run, is itself an installation token for a GitHub App. Its permissions are set by the permissions: key, and any action in the job can read it via github.token even if you never pass it.
18. A common trap follows from that. The workflow scope discussion in 42.10 applies here too: GITHUB_TOKEN deliberately cannot push changes to workflow files, and cannot trigger further workflow runs, to prevent infinite loops and privilege escalation.
19. The honest version, and it is a real trade-off. OIDC federation is more setup than pasting a token. For a one-person hobby project a short-lived fine-grained PAT is a reasonable choice. For anything with real secrets, the setup cost is paid once and removes a whole class of incident.

WORDS42.11.6 remember these#

Bearer credential — whoever holds it, is you — a token requiring no proof of possession beyond presenting it.

OAuth — letting an app act for you without your password — the delegation framework of RFC 6749, exchanging a user approval for a scoped token.

GitHub App — a robot with its own identity and badge — an installable actor with per-repository permissions and one-hour installation tokens.

Installation access token — an app’s short-lived key to one installation — a token valid for 1 hour, scoped to the repositories the app is installed on.

OIDC federation — proving who you are instead of holding a secret — exchanging a signed job description for short-lived cloud credentials.

Blast radius — how much damage one leak causes — the set of resources and the time window a compromised credential can affect.

42.12 Practical hardening for a developer’s own machine#

PLAIN42.12.1 in simple words#

1. Most security advice is too long to follow. Here is the short version that actually pays for itself.
2. Put a passphrase on every private key. It is the difference between a stolen laptop being embarrassing and being catastrophic.
3. Use the SSH agent so the passphrase costs you one typing per day, not one per push.
4. Make a different key on every machine. Never copy a private key between machines, not even your own.
5. The reason is simple. If you lose one machine, you remove one key. If you copied one key everywhere, you have to replace it everywhere at once, in a hurry, while panicking.
6. Look at your list of keys and tokens once or twice a year and delete the ones you cannot account for.
7. On any server you run yourself, turn password logins off entirely. Keys only.
8. Install a small program that bans addresses which repeatedly fail to log in. It is not clever, but it removes most of the noise.
9. Moving SSH to a port other than 22 stops the automated scanning almost completely. Be honest that this hides the door, it does not lock it.
10. And know in advance what you will do when a key or token leaks, because the first ten minutes decide how bad it gets.

PLAIN42.12.2 a picture in your head#

1. Think of the difference between a bicycle lock and a bicycle in a locked shed.
2. A passphrase on your key is the lock. Someone who takes the bicycle still cannot ride away immediately.
3. Separate keys per machine is owning three cheap bicycles instead of one expensive one. Losing one is annoying, not ruinous.
4. Disabling password login on your server is removing the letterbox that people keep pushing lock-picks through.
5. Banning repeat offenders is a doorman who remembers faces.
6. Changing the port is painting the shed to look like a hedge. Any determined person who walks the whole street still finds it.
7. That last one is worth doing anyway, not because it stops attackers, but because it stops your log file filling with thousands of pointless entries and hiding the one that matters.

Where this comparison breaks:

8. A thief must be physically present. An attacker scans the entire internet’s address space in hours, so “nobody would bother with my little server” is simply false. Everything is scanned, all the time, automatically.

PLAIN42.12.3 a worked example#

1. A complete setup for a new machine, in order, with real commands.

# 1. one key, this machine, with a passphrase
$ ssh-keygen -t ed25519 -C "you@laptop-2026"

# 2. keep it in the agent so you type it once
$ ssh-add -t 8h ~/.ssh/id_ed25519

# 3. check what the agent holds
$ ssh-add -l
256 SHA256:WH1E2gSLhG6FgMsha//GINoq... (ED25519)

# 4. add the public half to GitHub, then test
$ ssh -T git@github.com

2. A minimal safe ~/.ssh/config for that machine:

Host *
    AddKeysToAgent yes
    IdentitiesOnly yes
    ServerAliveInterval 60
    HashKnownHosts yes

3. On a server you own, the three lines in /etc/ssh/sshd_config that matter most:

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password

4. Always test in a second terminal before closing the first one. Locking yourself out of a remote machine is a rite of passage that is best skipped.
5. Adding a passphrase to a key that has none, without regenerating it:

$ ssh-keygen -p -f ~/.ssh/id_ed25519
Enter old passphrase:
Enter new passphrase:

6. Checking whether a key file is encrypted at all. An unencrypted OpenSSH private key contains the string none twice near the start of the base64, for the cipher and the key derivation function.

PLAIN42.12.4 what is really happening inside#

1. A passphrase does not scramble the whole key file with the passphrase directly. That would be weak, because people choose short passphrases.
2. Instead the passphrase is fed through a slow, deliberately expensive function to derive an encryption key. Then that derived key encrypts the private key.
3. Slow is the point. If each guess costs a fraction of a second instead of a microsecond, a guessing attack becomes hundreds of thousands of times more expensive.
4. When you type the passphrase, the same derivation runs, the file is decrypted in memory, and the plain key is handed to the agent.
5. From then until the agent is emptied or stopped, the unlocked key exists in that process’s memory and nowhere else on disk.
6. A ban tool such as fail2ban works by reading the authentication log, matching failure lines with a pattern, counting them per source address, and inserting a firewall rule when a threshold is crossed.
7. It is not intelligent and it does not need to be. Automated scanners do not adapt; they move to the next address.
8. Changing the port works for the same unglamorous reason. Mass scanners prioritise port 22 because that is where the yield is. A different port is still found by a full port scan, just far less often.

TECHNICAL42.12.5 the engineer’s version#

1. Key encryption in the OpenSSH format uses bcrypt_pbkdf, a password-based key derivation function built on bcrypt, introduced with the new private key format in OpenSSH 6.5, January 2014.
2. The work factor defaults to 16 rounds and is adjustable at generation time:

$ ssh-keygen -t ed25519 -a 100 -C "you@laptop"

3. -a 100 sets 100 KDF rounds. It makes unlocking take noticeably longer for you, and proportionally longer for anyone brute-forcing the file.
4. Since OpenSSH 7.8, August 2018, the OpenSSH format is the default for all key types, replacing the old PEM format which used a weak MD5-based derivation.
5. Server hardening, the directives that matter and what each removes:

6. Validate before restarting, every time:

$ sudo sshd -t
$ sudo systemctl reload ssh

7. fail2ban reads /var/log/auth.log on Debian-family systems or the journal on others. A typical jail:

[sshd]
enabled  = true
port     = 22
maxretry = 4
findtime = 600
bantime  = 3600

8. That bans an address for one hour after four failures within ten minutes. With PasswordAuthentication no already set, the practical benefit is log volume rather than security, which is still a real benefit.
9. sshguard and CrowdSec are alternatives; CrowdSec additionally shares reputation data across participants.
10. Changing the port. State the honest position clearly, because both exaggerations are common.
11. It is security through obscurity. It provides no protection against anyone who scans your ports, which any real attacker does.
12. It does reliably remove the overwhelming majority of automated noise, because mass scanners target 22 by default. Measured on typical internet- facing hosts, failed login attempts commonly drop by more than 95 percent.
13. So the correct framing is: it is a log hygiene measure with a security side effect, and it must never be the only measure.
14. Note also that ports below 1024 require root to bind, so a high port such as 2222 is bindable by an unprivileged process, which is a small argument against picking a high port on a shared machine.
15. Key rotation, as a routine rather than an emergency:

16. Leak response, and this is the part to rehearse. The order matters more than the speed.
17. For an SSH key: remove the public key from every account and server that has it. GitHub’s SSH key page shows a last-used date for each key, which is the fastest way to spot one you do not recognise.
18. For a token: revoke it in settings. Instant.
19. Then, in both cases, review what was done. Check the account security log, the organization audit log, and the repository’s recent pushes.
20. Look specifically for the standard persistence tricks: a newly added SSH key, a new deploy key, a new webhook, a new collaborator, a changed workflow file, or a new GitHub App installation.
21. Then rotate everything the compromised credential could read. A token with repo scope could read every secret ever committed to any repository it reached, including ones you deleted from the working tree but not from history.
22. Finally, write down what happened and how the credential got out. Almost every leak is one of four things: committed to a repository, pasted into a chat or ticket, printed in CI logs, or left on a machine that was lost.

WORDS42.12.6 remember these#

Passphrase — a secret that unlocks your key file — a user secret fed through a KDF to derive the key that encrypts the private key at rest.

bcrypt_pbkdf — the slow function guarding your key file — the KDF used by the OpenSSH private key format, with an adjustable rounds parameter.

fail2ban — a doorman that remembers repeat offenders — a log-scanning daemon inserting temporary firewall bans after repeated authentication failures.

Security through obscurity — hiding rather than locking — a measure that raises the cost of finding a service without raising the cost of attacking it.

Key rotation — replacing credentials on a schedule — periodic reissue and retirement, limiting the useful lifetime of an undetected compromise.

Persistence — what an attacker leaves behind — added keys, hooks, collaborators or workflow changes that survive the original credential being revoked.

42.13 A diagnosis table for git authentication failures#

PLAIN42.13.1 in simple words#

1. Almost every git authentication problem announces itself with one specific sentence.
2. If you learn to read those sentences, you stop guessing and start fixing.
3. There are three questions to ask in order, and they narrow it down fast.
4. First: how long did it take? Instant means somebody replied. Half a minute of silence means the network.
5. Second: did it ask for a user name or password? That means it wanted a credential and either had none or was told the one it had was wrong.
6. Third: did it mention SSH keys, or a repository name, or a scope? Each of those points at a different layer.
7. The most misleading message of all is Repository not found, because it usually does not mean the repository is missing.
8. It usually means your credential is not allowed to see it, and the server is refusing to confirm that it exists.

PLAIN42.13.2 a picture in your head#

1. Think of a doctor listening to a description of symptoms.
2. “It hurts when I press here” narrows things down enormously, far more than “I feel unwell”.
3. Error messages are the same. Permission denied (publickey) is “it hurts when I press here”. Push failed is “I feel unwell”.
4. So the first job is always to find the most specific line in the output, not the last line.
5. Git prints the summary last and the cause first. People read the last line and miss the answer three lines above it.

Where this comparison breaks:

6. A patient can be asked follow-up questions. A one-shot error message cannot, which is why the verbose flags exist.

PLAIN42.13.3 a worked example#

1. Get more information before guessing. Two flags do almost everything.

# for SSH remotes, ask ssh to narrate
$ GIT_SSH_COMMAND="ssh -vvv" git push origin main

# for HTTPS remotes, ask git to narrate
$ GIT_TRACE=1 GIT_CURL_VERBOSE=1 git push origin main

2. And the single most useful check for SSH, which tests only the connection and touches no repository:

$ ssh -T git@github.com
Hi USERNAME! You've successfully authenticated, but GitHub
does not provide shell access.

3. That message is a success. The words “does not provide shell access” alarm people every time. GitHub is not a shell server; it only runs git commands.
4. If that greets you by name, your key and your network are both fine, and any remaining problem is about a specific repository or a specific push.

PLAIN42.13.4 what is really happening inside#

1. The reason one problem produces several different messages is that several independent programs are involved, each with its own vocabulary.
2. Your kernel produces Connection refused and Connection timed out. These are TCP-level outcomes and mention no names.
3. ssh produces Permission denied (publickey) and Host key verification failed. These are SSH-level and mention keys.
4. curl inside git produces HTTP status codes and TLS errors.
5. GitHub’s own server produces every line beginning with remote:. Those are the only messages that reflect a decision by GitHub rather than a failure of plumbing.
6. So the prefix tells you the layer. A line starting with remote: came from the far end and means you got there. A line starting with ssh: means you did not.

TECHNICAL42.13.5 the engineer’s version#

1. The table. Each row is an exact string you will see.

2. Now each one in full, with the action to take.
3. git@github.com: Permission denied (publickey). SSH connected and the server rejected every key offered. Causes, in order of likelihood: the public key was never added to the account; the agent is empty; the wrong key is being offered; IdentityFile points at the public key by mistake; on your own server, wrong permissions on ~/.ssh or authorized_keys. Check with ssh -T git@github.com and ssh-add -l. On a server you control, read the server log, which is the only place bad ownership or modes appears.
4. remote: Invalid username or password. or remote: Support for password authentication was removed on August 13, 2021. The HTTPS credential is not valid. Either it is an account password, which has not worked since that date, or the token expired, was revoked, or was for a different account. Clear the cached credential with git credential reject or the platform keychain command, then retry so you are prompted afresh.
5. ! [remote rejected] main -> main (refusing to allow a Personal Access Token to create or update workflow ... without workflow scope) The credential is valid. The push touched .github/workflows/. Add the workflow scope to the token, or the Workflows write permission to a fine-grained token, then push again. Nothing was written; the branch is unchanged.
6. Host key verification failed. The server’s key does not match the saved one, or you declined the prompt. Compare the printed fingerprint against the operator’s published one before doing anything else. If and only if it matches, remove the stale entry with ssh-keygen -R github.com and reconnect. Never delete the file wholesale to make a warning go away.
7. ssh: connect to host github.com port 22: Connection timed out Nothing replied. Port 22 is filtered on this network, or the path is broken. Test 443 as a control. Fix with an HTTPS remote or the ssh.github.com port 443 configuration in 42.7. Expect roughly 15 to 130 seconds before this message appears, depending on the system.
8. ssh: connect to host X port 22: Connection refused The opposite of the above and far friendlier. Something answered and said no. The host is reachable and nothing is listening on 22. Either sshd is not running, or it listens on another port. Measured in milliseconds, not seconds.
9. remote: Repository not found. followed by fatal: repository ... not found. Read this as “not found for you”. Causes: a typo in the owner or repository name; the repository is private and your credential lacks access; you are authenticated as a different account than you think; a classic token without repo scope, which cannot see private repositories at all; an organization with SAML single sign-on where the token has not been authorized for that organization.
10. Why 404 rather than 403? Because returning 403 would confirm the repository exists. GitHub deliberately returns “not found” to anyone not permitted to know. We measured this directly in 42.8: an anonymous request for a nonexistent repository returned 401, not 404.
11. ! [rejected] main -> main (non-fast-forward) or (fetch first). Not an authentication problem at all. Your branch is behind the remote. git fetch then rebase or merge. If you push after a rebase, use --force-with-lease, never plain --force.
12. send-pack: unexpected disconnect while reading sideband packet The connection died while git was reading the server’s response. This is the error the reader saw twice during a network outage.
13. It deserves a paragraph of its own because it is the most dangerous message in this list, and for a reason that is not obvious.
14. It tells you the response was lost. It tells you nothing about whether the write landed. The server may have applied your push completely and then failed to tell you.
15. So never assume either outcome. Re-query the actual state: git ls-remote origin refs/heads/main and compare the hash against your local commit.
16. Do not trust origin/main in your local repository either. That is a remote-tracking reference, which is a cached copy of what the remote said last time you successfully spoke to it. During an outage it is stale by definition.
17. Two more you will meet:
18. Too many authentication failures The agent offered more keys than the server’s MaxAuthTries allows before you got to the right one. Fix with IdentitiesOnly yes and an explicit IdentityFile for that host.
19. sign_and_send_pubkey: signing failed: agent refused operation The agent has the public key but cannot use the private key. Usually a key added from a file that has since changed, or a hardware key that needs a touch. Run ssh-add -D then re-add.
20. A general ordering rule that resolves most confusion: read the output from the top, find the first line that names a cause, and ignore the summary at the bottom. error: failed to push some refs is never the reason.

WORDS42.13.6 remember these#

remote: prefix — a line that came from the server — output relayed from the far end, proving the connection and authentication both worked.

Sideband — the side channel carrying server messages — the multiplexed stream in the pack protocol used for progress and error text.

Remote-tracking reference — your cached note of what the server said — refs/remotes/origin/main, updated only on a successful fetch or push.

--force-with-lease — force push, but safely — a push that refuses if the remote moved since your last fetch, unlike plain --force.

ls-remote — ask the server what it actually has — a command querying live references without touching your working tree.

SAML SSO authorization — an extra approval step for org access — an organization requirement that a token be separately authorized before it can reach that organization’s repositories.

42.98 Common wrong ideas#

1. Wrong: SSH and HTTPS reach different copies of the repository, so switching loses work. Right: it is one repository. Only the road changes. Switching is one git remote set-url and nothing is lost.
2. Wrong: the private key is sent to the server, so publishing the public key is risky too. Right: the private key never moves. The server sends data, your machine signs it, and only the signature crosses the network. The public key is designed to be pasted into web forms on purpose.
3. Wrong: Permission denied (publickey) means the key is wrong. Right: it also appears when ~/.ssh or authorized_keys has loose permissions, when the agent is empty, and when too many wrong keys were offered first. Only the server log distinguishes them.
4. Wrong: the host key warning is a nuisance, so delete known_hosts, and anyway a change always means an attack. Right: both halves are wrong. It is the one warning that detects an interceptor, so compare the fingerprint against a published one and remove only the stale entry. And GitHub legitimately rotated its RSA host key at about 05:00 UTC on 24 March 2023.
5. Wrong: port 22 being blocked means GitHub is down, and a blocked port fails immediately. Right: on the machine used for this chapter, port 22 timed out after 30.057 seconds while port 443 connected in 0.021 seconds to the same host. A DROP sends nothing back, so your machine retries with doubling gaps and hangs. Only a REJECT fails fast, with Connection refused.
6. Wrong: a token is just a password with extra steps. Right: a token can be scoped to specific actions, expire on a date, and be revoked alone. A password can do none of those things, which is why GitHub stopped accepting passwords for git on 13 August 2021.
7. Wrong: the repo scope includes everything, so a push cannot be refused for permissions. Right: repo does not include workflow. A push touching .github/workflows/ is refused without it, even though authentication succeeded and every object transferred.
8. Wrong: remote rejected means the network dropped the push. Right: the word remote: proves you reached the server and it answered in sentences. That is a policy decision, not a transport failure.
9. Wrong: Repository not found means the repository does not exist. Right: it usually means your credential may not see it. GitHub returns “not found” rather than “forbidden” so it does not confirm that a private repository exists.
10. Wrong: changing the SSH port secures a server, and agent forwarding is just a convenience toggle. Right: a different port only removes automated scanning noise, since a real attacker scans all ports. And root on the machine you forwarded to can use your agent silently, against every server your keys open, for as long as you stay connected. Prefer ProxyJump.

42.99 Chapter summary in 20 lines#

1. Git borrows two transports: SSH on TCP 22 and HTTPS on TCP 443. The repository is identical either way; only the road and the credential differ.
2. Over SSH, git logs in as the user git and runs one command remotely, git-upload-pack or git-receive-pack. Over HTTPS it makes one GET and one POST.
3. SSH was written by Tatu Ylonen at Helsinki University of Technology in 1995 after a password-sniffing attack, replacing telnet, rlogin and rsh, which sent passwords in clear text.
4. SSH-2 replaced the structurally flawed SSH-1 and is specified in RFC 4251 to RFC 4254. OpenSSH forked from Bjorn Gronvall’s OSSH on 26 September 1999 and shipped with OpenBSD 2.6 on 1 December 1999.
5. SSH-2 has three layers: transport for encryption and host key verification, user authentication, and connection for multiplexed channels.
6. A key pair splits into a private key that signs and a public key that only verifies. Publishing the public half is safe; the private half never leaves your machine.
7. Authentication is challenge-response: the client signs a block containing the session identifier, so an old signature cannot be replayed and a man-in-the-middle cannot forward yours.
8. Ed25519 is the current recommendation: deterministic signatures, 32-byte public keys, published parameter rationale, RFC 8709. RSA 2048 is the floor and 4096 is common. DSA is removed everywhere.
9. authorized_keys is a plain text file of public keys. Wrong permissions make the server ignore it and return Permission denied (publickey) with no explanation to the client.
10. Options in front of a key restrict it: command=, from=, no-port-forwarding, and restrict for everything at once. Deploy keys are the per-repository variant.
11. known_hosts is the other half nobody reads. The server proves itself first, and the first connection is trust on first use, which is the model’s honest weakness.
12. GitHub rotated its RSA host key at about 05:00 UTC on 24 March 2023 after the private key was briefly exposed publicly. That is what a legitimate change looks like.
13. The agent holds unlocked keys and signs on request, so the key never leaves its memory. Agent forwarding lets root on the remote host use your keys and should be replaced by ProxyJump.
14. Port 22 is commonly filtered outbound because it is heavily scanned and useful for exfiltration, while 443 must stay open or the web breaks. A DROP hangs for tens of seconds; a REJECT refuses in milliseconds.
15. The two fixes are switching the remote to HTTPS, or using GitHub’s SSH endpoint ssh.github.com on port 443 with a four-line ~/.ssh/config block.
16. GitHub removed password authentication for git operations on 13 August 2021. Credential helpers now store tokens, in the OS keychain if you are careful and in plain text if you use store.
17. A token can be scoped, can expire and can be revoked alone. Its prefix, ghp_ or github_pat_, exists so that secret scanning can find it, and push protection blocks it before it ever leaks.
18. A scope is a named permission checked at the moment of the operation. Pushing a file under .github/workflows/ needs workflow, because changing CI changes what runs on GitHub’s runners with the repository’s secrets.
19. Distinguish the three “push failed” cases by evidence: silence and a long wait is network, a credential prompt or Invalid username is authentication, and ! [remote rejected] with a sentence is authorization.
20. Four identities with four blast radii: SSH keys, personal access tokens, OAuth grants and GitHub Apps. CI should hold no long-lived secret at all and use short-lived OIDC-federated credentials instead.