Secure software development is one of the biggest concerns for enterprise security today. Unfortunately, many enterprises are continuously playing catch-up from original poor software design.
In this special full-length video presentation, security luminary Marcus Ranum discusses the consequences of bad Internet architecture decisions and poor software design, and what can be done to ensure this does not happen in the future.
Editor’s note: This presentation contains some strong language. View discretion is advised.
Table of contents:
- 3:01 - File Transfer Protocol
- 6:22 - The basics of network sockets
- 7:04 - How FTP works
- 9:09 - Internet firewalls
- 13:13 - More on sockets
- 15:13 - Partial socket tables
- 25:24 - The state of statelessness
- 26:32 - Software frameworks
- 29:06 - How load balancers work
- 30:14 - Achieving TCP/IP protocol
- 31:58 - Session management
- 34:51 - Thinking ahead
About the presenter:
Marcus Ranum is Chief of Security for Tenable Security, Inc.
Read the full transcript from this video below:
Marcus Ranum on the consequences of poor software design
Marcus Ranum: What I thought I would do today is talk about something that's a little bit
out of left field, and by the time I'm done, you may be wondering, did he
have a point. I do have a point. I'm just not entirely sure what it is.
What I want to do is talk about some of the sort of odd historical side effects
that we've seen and how they've impacted computer security of, I can't
really call them bad design decisions but, shortsighted design decisions
that were made under fire at the point where people were trying to actually
accomplish something, the side effects, downstream, of those short-term
decisions and how they've affected all of us. Again, when I started thinking about
computer security in this way that was when I started getting depressed
because I realized that what we're mostly dealing with in security is we're
dealing with the downstream consequences of 10-year-old mistakes.
I know that some of you are stressed out right now. You're worried about,
"How do I talk my corporate management out of the idea of moving our online
banking application to Facebook," or outsourcing our bank to Nigeria or
something like that. It already is? Well . . . doing something incredibly
stupid. But, the problem is, what we don't realize a lot of the time, is
we're trying to stop horribly stupid things from happening but we ignore
the fact that most of the problems we're dealing with, right now, are the
results of 10-year-old stupid or 15-year-old or 20-year-old stupid. The
stupid that is happening today hasn't even kicked in yet, right? So, in
another 10 or 15 years, you're really going to be screwed. So, that's the
cheerful part of the talk.
What I wanted to do was try to illustrate how this defective software
creation has interesting side effects. A certain amount of this was
inspired by my, well, now she's my ex-girlfriend, but this girl that I was
dating for a while was a biologist. We were talking about evolution and I
made a comment about how computer security needs to adopt a more biological
model because computer security people say this fairly often. She looked at
me and said that's the "dumbest thing I've ever heard of," which kind of
hauled me up short. I said, "What do you mean?" because she never spoke
lightly about things like that. Then she proceeded to explain to me that
evolution is the most wasteful algorithm that's ever been invented. It's
actually also a very good one. If you have 13 billion years in which to
evolve your system, it will get pretty decent. That appears to be what
we're doing with computer systems. Right?
So, if you actually evolve things, you get horrible side effects and, well,
you'll see. So, in the beginning, before there was an Internet, before
there was TCP/IP, before there was a lot of the stuff that we're dealing
with nowadays, there was FTP. It turns out that FTP appears . . . The file
transfer protocol is what I'm talking about here. The FTP protocol, the
FTP, appears to have been around sort of since the dawn of computing. It
seems that the first thing people did when they thought of how to get data
back and forth by computers, instead of carrying shoeboxes full of
cardboard punch cards, the first thing they did was FTP. I've asked old
computer programmers constantly. I pestered poor Tom Van Vleck about it. I
dug him up and asked him and he said, "Shucks, that's really a good
question. I don't know." Well, if Tom doesn't know, nobody knows. He said,
"You're probably right." I said, "Who was it?" He said, "Nobody knows." He
But, this FTP thing was kind of one of the first applications that was ever
built that did anything between two computers and, in fact, the first
generations of email were carried via FTP. The way that email used to go
back and forth was essentially a gigantic batch job that took all of your
email messages as files and FTPed them to another machine and then stuck
them into the bottom of the input queue for the mail system and that's how
Internet email was invented. That later got enhanced into a whole different
protocol called, SMTP, Simple Mail Transfer Protocol, which I also don't
have a date for but that's also protocol that's been around and that we're
dealing with for a long time. One of the consequences of SMTP's choice of
not authenticating anything, which was actually a consequence of the design
of email being based on FTP, because you couldn't authenticate it if it's
just files that you're copying back and forth, is that we have a whole lot
Circa 1971, we had FTP and in 1976, there was a thing called Host to Host
Protocol. Then Host to Host Protocol evolved into another thing called NCP,
which was the Network Control Protocol. All Host to Host Protocol was a
simple packet layout, a simple messaging format that was used so that
computer A could talk to computer B and it did. Actually, this was a huge
innovation at the time. Computer A and computer B didn't actually have to
be the same kind of computer because they're using the standard messaging
format for sending stuff back and forth. You could actually have a machine
from Digital, which just started as a business, at that time, and a machine
from IBM that could actually talk to each other and this is a huge step
forward. So, Host to Host Protocol and NCP was kind of the first standard
protocol, a very cool idea. Right? We're going to have a standard protocol.
Different systems can talk. We're going to have a standard FTP protocol.
Different systems can transfer files. This is really cool. I can actually
get data from a mainframe to someplace else without having to cut tape and
walking back and forth and then run it through a tape translation process.
It's very cool. Right? So, this is really the first use of the Internet,
essentially was FTP.
Now, just a little bit of introductory stuff. I know most of you guys
already know this but there are these things called sockets and sockets are
a rendezvous point on the Internet. This is relevant. The way a socket
works is it's just an abstraction in data space that allows you to transfer
data. When somebody wants to transmit data, you've got a problem. You've
need to have someone that's ready to receive data. So, a socket, you
basically have a listener that listens on a port and says, "I'm willing to
receive data," and then someone connects to it and then data gets
transmitted and when you're done it gets disconnected and that's how all
the transactions work on any kind of public network that's worth talking
about. That's how they're all done.
NCP did this in 1976, 1977. Now, what FTP does is a slightly fancier
version built on top of that basic protocol for that basic paradigm of how
you get data back and forth. The way it works is little bit bizarre. What
happens is the FTP server is listening on a port. The FTP client connects
to it. "We want to transfer some files.' The client then sends a user, "I
would like to log in." The server asked for a password. The client sends a
password. The client then does a very funny thing. Everything is kind of
normal up to this point and then something very strange happens. What
happens is the client creates a port of its own, kind of at random, and
tells the server, "Transfer this particular file to the port that I just
created and here's my IP address and port number of this socket that I'm
now listening to." That was really horrible grammar but you got the idea.
And the server then connects back and transfers the data. What the hell?
The server then tells the client, "I'm done transferring it," because the
client's apparently not smart enough to figure that out and drops the
connection and everything is finished. That's how it's done.
Now, why did it do it this way? It did it this way for a very stupid
reason. It did it because NCP could only carry data in one direction on a
socket. You could only send stuff in one direction. So, what was actually
happening in the first versions of the FTP protocol, I won't go back and
torture you by redoing this, is you actually had two connections. You had
the server connected to the client, twice, I’m sorry, the client
connected to the server, twice, and the client's commands to the server one-
on-one connections, and the servers replies, one on the other, and they
needed a certain action which was this bizarre hunky monkey dance in order
to get your data back and forth, still preserving the uni directionality of
data transmission over a socket.
Now, that was the first thing that was fixed when the next-generation
protocol was invented, which was TCP, which was essentially the next-
generation of the protocol suite that was being used for the first-
generation ARPANET and then that whole thing with the hunky monkey dance
was no longer necessary but it remained in the code because fixing it would
Now, why does this matter? It matters because of Internet firewalls. So, if
you zoom back in time to about 1987, 1988, people were starting to put
systems on networks that they didn't own or didn't entirely own. So, you'd
actually have a hospital, where I was working at the time that this was
happening, you have a hospital that was putting hospital systems onto a
network that was connected to the Internet, that was connected to the
University, that was then connected to who knows what and nobody asked
anything about this. It just sort of happened, until you noticed people
logging into the CAT scanner machine, from strange places and we're all
kind of shocked. That was actually my introduction to security.
But, the problem was that we wanted to build some sort of system to prevent
this kind of thing from happening, to redefine the boundaries of our
networks. We called those firewalls in those days. Everything would've been
just fine because the first-generation of routers that we had available to
us had the ability to block traffic based on its origin by watching for the
established flag in the TCP sequence numbers, except for FTP.
FTP did this bizarre thing where it made a call back into your network from
the outside and we couldn't fix that with just a plain old router. We
needed to do something else and, as you know, in computer science, doing
something else always involves buying another box. So, we put another box
on the network and the box became a firewall and it became, essentially, an
FTP mediator device. It would sit there in the middle and I'm not going to
torture you by walking you through the protocol handoffs that happened here
but the client talks to the firewall, the firewall talks to the server, the
server talks to the firewall, the firewall talks to the client and it just
works fine and is actually quite fast and you don't notice any of this
happening. This kind of technology is still in a tremendous amount of wide
use and the reason is because it's useful to have a box sitting between
your network and someplace else, that you can load spam filtering and
billing and quality of service. You can load all kinds of stuff into it.
But, the point of this is, we would not have needed the firewalls. The
firewalls might not have happened at all if FTP hadn't had those rather
bizarre little quirks in its first-generation of code.
Now, in 1991, the first commercial firewall, which I happened to code and
sell, sold for $175,000, of which I saw my regular paycheck. Yes. By 1994,
the numbers are a little bit hard. This is kind of extrapolated from
some Gartner stuff, from years ago because the terms of what a firewall is
has changed but the industry skyrocketed. Right? It went from a couple of
vendors with very small sales up to ginormous sales and, depending on what
you want to call a firewall, if you want to call it "network edge defense",
we're looking at spending literally billions of dollars over the course of
the computer security industry over boxes that essentially fix FTP.
The funny thing is, that if you think about it, a competent programmer
would've been able to fix FTP in about two hours. Around 1975, we wouldn't
have needed any of that stuff. Well, it's good for some of us because I
wrote firewalls for a living, for a couple of years, and it was really nice
and so, thank you all. Everyone who's ever bought a firewall has helped
keep me alive and happy, so I appreciate that. But, we spent a huge amount
of money over something that, ultimately, was a trivially fixable problem
and that's interesting. That's really odd. Right? The funny thing is we
spent all this money on firewalls and the problem is still there. We didn't
actually fix it. We just simply made it less painful. Why didn't we just
actually fix it? I don't know.
Let's look at another example. That was an easy one. That was a painless
one. This is the one that really hurts and this is the one that's going to
really hurt. Okay? So, a little more about sockets. In the early 1990s, all
of the implementation of the Internet, pretty much, except for a few [VACs]
for those of you who remember those, were running on derivatives of things
running the BSD flavor of UNIX. There were a few System 5 things, but
System 5 in early 1990 had as its IP stack, basically the Berkeley-derived
Do you remember back in the days when you used to buy Windows or DOS and it
didn't come with networking? You had to buy Chameleon or I can't even
remember any of these companies anymore but you had to pay money to get an
IP stack in Windows because Bill Gates was a big visionary about
Okay. Well, Bill Gates was almost a big visionary about networking.
Seriously, Bill's a very smart man because that's why he owns so much
stuff, but that was really amazing. You look back in time and you wonder
how the heck could anybody buy an operating system that didn't have
networking built into it? It was treated as a value add. Was Scott McNealy
right, that the network is the computer? That's a scary thought. Anyway,
speaking of getting it wrong and Scott McNealy, those two go together.
So, the important point of this is that the network layer, in UNIX, at this
time, in the early 1990s was all running the same code base. That's
important. Why is it important? There was a stupid little programmer's
kludge in Berkeley UNIX, because the guys who coded this did this as their
undergraduate research project and they didn't really think very hard about
what they were doing. They didn't realize that they were building the
infrastructure for the future of the planet, which, if you talk to all the
cyber war jar heads they'll tell you that this is the battlefield of the
future. These guys didn't realize that they were coding the battlefield of
the future. Right?
So, they had this thing called a partial socket table. Now, when you get
this connection on a socket in TCP, what you basically do is you send a
packet saying, "I would like to make a connection to you." You get back a
packet from the server saying, "I would like you to like to make a
connection with me. Here's a kind of partially-formed handle for a
connection and if you really want to connect to me, you can use this handle
for the rest of your data," and the client then sends more packets that
have got that sequence number on the same port pair, and everything's hunky-
dory and you've got a connection. But, you have this brief period of time
where the connection is not really live. It's after that second packet goes
out. The connection's partially live because the server has essentially
acknowledged, "Okay. I got stuff from you and," and the client hasn't
started to really send or do any data yet and so, you've got this kind of
intermediate state that usually lasts for about a millisecond, at most.
Even on a slow network, it lasts for less than a millisecond, in general,
unless you're on dial-up or something like that.
So, what the programmers did was they went, "Well, that doesn't happen very
often and it doesn't stay around for very long," so they have a system
socket table, which is all the active connections and they made his partial
socket table which is these connections that are kind of in progress and
they pulled a number out of their backside and they made it 12. So, there
are 12 of these entries that you could have. It never filled up, under any
practical terms until somebody actually invented the SYN flood attack by,
"What if I just send a whole bunch of connection request packets and don't
carry out any further communication with you." What happens is the
operating system hangs. Whoops. That was actually an early denial of
service attack but that wasn't really being popularized around this time.
The problem was you have his partial set of connections and it would run
out and the operating system didn't do a very good job of mediating these
partial connections. So you'd get the server that wants to listen to the
incoming connection. You put it on this temporary flag over here and then
when it's done, you drop it over the system table.
Both of those tables had fixed sizes and the only way to make the system
have more, was to recompile the operating system from scratch, which, if
you had AT&T source license or Berkeley source license, you could do it.
But, if you are most normal people, you just used whatever number the
vendors gave you, which meant that if you are running on [DEC ULTRIX] it
was 2048, if it was Sun, it was 4000, or whatever. But, the point is if you
had a machine that was doing a lot of networking, you could run out and
you'd start getting these bizarre failures because something would try to
open a network connection and it would run out of network connection space,
handles, handle space in the system which is hanging, basically saying,
"insufficient sockets." That would be a big problem. I don't know if any of
you were ever system administrators and had that experience in those days.
I had a couple of times. People would come running into your office going,
"The system has run out of sockets," and you'd go, "What the hell should I
do?" My friend Fred used to have a cardboard box in his office labeled
"sockets" any say, "I've got more. I'll go install some in a little while.
Just be patient." The problem would usually go away anyway.
But, in 1995, some versions of UNIX would crash and burn if they got too
many of these connections. So, what was another big thing that was
happening in 1995? It was happening at a place called CERN. Right? Some
clowns wrote this World Wide Web, HTTP crap and we've been dealing with it
ever since. The problem was, Berners-Lee and those guys were not systems
programmers. They were application programmers and of course we system
programmers always laugh at the application programmers. That's a
tradition. But, the app guys did not build a kernel from scratch and so
when they ran out of socket space, because they were building these
inefficient servers that were making lots and lots of requests, they
decided they were going to make the protocol stateless which meant that it
made the connections last for very short periods of time, which meant that
you'd run into these little connection storms but you wouldn't run out of
sockets. That's the only reason they did. That's the whole reason they did
So, just a little refresh, when you go to a webpage and you access
something, you go to HTTP something or other, your client machine or your
browser makes a socket connection and the server asks for that URL and
pulls one piece of data and disconnects. Usually what that piece of data
contains now is 87 embedded pieces of Flash and Gifs and all kinds of other
assorted animated crap. But, then your browser goes and makes each one of
those as a separate connection request and pulls down the data and then
assembles it into something that it can stick on your screen. So, each time
you go to a particular webpage, you could be actually triggering 5 or 6 or
10 or whatever many connections and then assembling that into a coherent
It got worse. This was slow. Well, one of the reasons it's slow was
because, yes, if I want to connect to your site, and get a page, if this
had been devised by people who actually knew anything about system
programming, I would make a connection and I would request, "Here's the
five pages that I would want," and then you would send me the five pages
that I want with some kind of delimiter between them, all in one single
transaction. The first thing would probably fit inside a single packet. The
rest of the stuff would all be globbed together. We could compress it and
take advantage of it and then shut the session up and it would just work.
That's not how the guys from CERN did it because they didn't know how to
program that well.
It also turned out that doing these little short life connections was slow.
So, instead of going, "Hey, let's fix our stupid protocol,' they said,
'Let's make lots of connections in parallel." Remember? So, now, a browser,
when you go to a site, what usually happens is you do one connection and
then your browser does 10. So, you're actually making a whole bunch of
connections to the site and the browser gets a whole lot faster and the
servers get the snot whacked out of it.
I don't remember how many of you were around, around 1995, doing any kind of
system support stuff. The game in 1995 was, "How many web hits per minute
can your server handle before it dies?" That was kind of the benchmark. I
mean, if you were a tough guy . . . I remember where the Berkeley guys,
actually the guys from BSDI, were actually able to get something like
10,000 hits per minute out of a 486/66, running with the website entirely
cached in RAM and Microsoft had a big problem. They had a big internal
effort to get the speed of Microsoft IIS up to the point where it was twice
as fast as Berkeley, by cheating, because they were embarrassed because
they could only handle like 2000 web hits per minute, or something like
that. So, this was a big deal.
Now, the point is, in order to fix the results of having those multiple
parallel connections happening, they fixed the problem that made Berners-
Lee and his guys code HTTP stateless in the first place. Right? Because now
a Web server was getting thousands of connections per second, tens of
thousands of connections per second. We can't even talk about connections
per second, for a site like Facebook. Right? Because there's no actual
single thing you connect to any more. Its clusters of zillions of
connections happening per second. The whole metric is gone because the
whole problem of connections per second has been completely solved into
irrelevance by technology, ever since. But, the statelessness of HTTP
remains. So, we fixed it.
Here's the point. The original reason for doing these short life
connections, the reason HTTP was stateless, was actually solved a year
before HTTP became a popular protocol. The whole web explosion happened in
'95, '96. By '96 it was all done. It was a done deal, right? The industry
looked at the web and went, "That's the future of electronic commerce,"
and, suddenly, it was, which was really an amazing thing. But, the reason
for making those short-lived HTTP connections was solved within months of
the invention of HTTP in the first place. All Berners-Lee and his guys
would've had to do to look really smart would've been to have done nothing
for a couple of months, but that's not what they did.
So, Internet commerce worked and here's the problem. Statelessness, what
kind of idiot wants to do commerce in a stateless manner? Commerce, I say
this in New York, the city of commerce. Anybody from New York can tell you that commerce
is state. That's all it is. All money is is state. I have the state of
having a dollar. You don't. I can give you that state. I can transfer it.
Everything we do in computing that is interesting and/or useful and
especially, that makes us money, or loses us money, involve state. Right?
Because money is nothing but state.
So, what the guys who built the first generation of the Internet, simply
because of the socket's table size problem, was they left the state out of
the first version and they convinced everybody that this was a good thing.
Remember back in those days? Remember also, a few people, if you were
security people, you kind of went, "Hey, it would be cool if this web stuff
did authentication, maybe even encryption." Remember? It didn't do
encryption or authentication. "It would be cool if we added that." And
there are people who went, "No. Don't add that to our web protocol because
this information wants to be free." I don't know if you remember those very
short-lived cyber hippies. We took them all out and we schooled them behind
the woodshed and that was the end of that. Actually, we gave them stock
options and now we mow their lawns for a living, is really what happened.
Right? All those cyber hippies who are now driving around in their
Ferraris, they all understand the value of commerce.
But, here's the problem. Everything you want in the web, nowadays, has to
hold state, everything. Secure Socket Layer holds state. Secure Socket
Layer is, in fact, nothing but a way of injecting state into a stateless
protocol. I remind you, the purpose of TCP/IP, is that it holds state.
Right? It keeps connection. So, essentially, we had something that does
exactly what we wanted and we went to this huge circumnavigation to re-
implement the thing we already had, only not as good, which is really quite
So, we've got shopping carts. We've got website logins. Every single
website keeps track of what you see. Everything on the Internet that you
would possibly want to do, short of just looking at pornographic gifts,
unless you want to see the same one over and over again because the site
doesn't keep state, involves keeping state. I mean, if you want to see the
picture of the day, and not see it over and over again, it's going to keep
state about what you've seen. Right? So, this is a big problem.
Now, it gets worse. Again, remember, this is all because somebody, whose
name is not necessarily Sir Tim Berners-Lee, didn't go back and fix a
little bit of code that would've taken a couple of hours, probably, to fix.
Every single software framework that every programmer is programming in
today, has a different state model. Why? Because it was written by
programmers and they're not going to all use the same thing, except for the
guys at Microsoft who do use the same thing because that's how Microsoft
makes their employees behave. But, PHP, Ruby, Ajax, every single online banking apps and
every single thing that you're working with is re-injecting state and is
doing it differently.
Some are doing it by encoding it in the URLs. Some are doing it by encoding
a special cookie. Some of them are actually doing, get this, they're doing
persistence. The idea is that if I want to do a bunch of transactions with
you, I connect with you. I authenticate myself. I do a bunch of
transactions and when I'm done I disconnect. Then you know I'm done, which
is how anybody but a retard would've coded this thing in the first place.
Every programmer whose writing any kind of application that does anything worth doing
in the first place has to learn one of these different state management
technologies that's appropriate for the web application framework that
Now, if you were to try to total up the sheer amount of lifeblood of
software engineers that we're spilling on the ground in re-implementing
state tracking, for HTTP . . . We are burning lives. They're insignificant
lives. They're just programmers. But we are wasting people's lives dealing
with this problem, which we actually don't need to have, which is
absolutely fantastic. It gets worse. It gets worse.
The leaving state out, because of the socket table size . . . it's all got
to do with the socket table size. Right? We have to have fancier ways of
dealing with state as we evolve things forward. Right? If you have a
website that's in one box, you can keep all the state in that one box and
it's going to work. It's going to be okay. You can have it all on the
server side stuff. But, what if your website gets so big, if you're doing
so many transactions per second that you need 50 boxes in an array or I
don't even know how many boxes Google's actually consisting of. I would
find that an interesting question. But, you've got thousands of computers
that are trying to act like a single computer. How do you preserve state
across those? You need entire new layers of state generating stuff to put
the state back that we could've had in the first place. Right?
That's why you've got these load balancers. A load balancer makes F5 very
happy, but a load balancer's just another box that sits in front of your
Web server that simulates state for all the machines behind it, so that the
application's frameworks won't break and if you actually look at the code
for something like an F5, it's absolutely horrendous because inside there,
they understand. They parse all the web traffic, which means they've got to
use some pretty nasty silicon to handle the stuff with the kind of data
rates that we're talking about. They parse all the traffic so they can
figure out what kind of state preservation mechanisms are in use and
simulate it. So, they simulate it on the front end and they fake it on the
backend by gluing the traffic so it always goes to the same place, which,
if you think about it, is really horrendous.
Again, it's making the guys at F5 happy. It's giving them something useful
to do with their lives instead of cleaning up birds and dying fishes in the
Gulf, or something like that. It's keeping them from having to have a real
job, but you're spending a lot of money for these things to re-inject state
that comes for free with TCP/IP, if Berners-Lee had just used it.
So, the point here is in order to really achieve the capability that our
underlying software has always had . . . That's what it's for. TCP/IP is
nothing but a tool for preserving state on connections between two machines
in an abstract addressing space. Right? We're spending hundreds of
thousands of coder hours every year. Were spending millions of dollars,
tens of millions of dollars, maybe even hundreds of millions of dollars on
devices to re-inject that state back into a network that we've made
stateless because of one decision.
So, the other part that sucks, unless your security guy and you, again,
need something to do for a living. Because of all the different ways of re-
injecting state back into the network protocols, we've got state injection
failures. We've got opportunities for the hackers to misbehave. If HTTP had
been designed by, I won't say smart people because I know they're smart
but, forward-thinking people, you would issue a connection to your server,
you would set up an encrypted link, if you needed one, negotiate an
encrypted link, pass an authentication credential over the encrypted link,
keep the connection up until you're doing stuff and when you're done doing
stuff you tear the encrypted link down and that's the way that the server
can know that you're done and I want my shopping cart cleaned up or my
transaction to complete or to delete my stuff or whatever it was that I was
doing, I'm now done. Right? Otherwise I keep the connection nailed up and
then the problem that we would've had is that website would've had to
manage millions of connections simultaneously, which is a problem that's
already solved. Websites know how to manage millions of connections
So, we wouldn't have had any of these problems with session hijacking,
cross site scripting, cookie injection or any of these things, all of which
are attacks that rely on the premise that, when I'm trying to connect to
his site, I connect to his site over and over again and there's any number
of places where somebody can redirect one of those connections or fool me
into connecting someplace else. That's what all these injection attacks
are. If I actually just connected to the site, did my thing, stayed up and
then just disconnected, we wouldn't have any of these problems.
So, what I'm doing, just to make it absolutely crystal clear is I'm just
taking this gigantic heaping pile of blame for this huge amount of suck and
I'm just pointing it at Tim Berners-Lee and saying, "If you had your
thinking cap on..." Well, I'm being a little bit facetious about it because
the truth is he had no way of knowing he was going to succeed. I'm
perfectly sure that I could go back in time and, after I stopped beating
him, and tell him that he was going to succeed beyond his wildest dreams
and "You're going to be knighted and you're going to be a multimillionaire
and you're going to win the Finnish equivalent of the Nobel Prize for this,
so don't get too angry, but could you just fix this stateless thing," he
might've listened to me. But, this is all stuff that could've just been
fixed. It would've taken eight or nine hours, maybe, of heavy coding, not
even heavy coding. It actually would've involved leaving a bunch of stuff
out rather than putting stuff in. So, mostly, it would've been, "Let's just
go and rip out the stuff where it drops the connection and reconnects and
we'll just leave the connection up." I mean, you're literally talking a
couple of hours to find the pieces of code that you need to delete and then
some testing, which would've been nice.
So, the point of all of this, if I could be said to have a point, is we
really suck at writing software, we humans, we collectively and I'm a
former software engineer so I'm taking my share of blame here too. We
really suck at producing software. We are producing stuff that is more
complicated than we humans seem to know how to deal with effectively.
If you're interested in this, there's a huge body of literature in
industrial accident analysis that just kind of drills into what happens to
the probability of failure when you have humans doing increasingly complex
things and then things that are interconnected. Probably the quintessential
book on this, if you're interested, is Charlie Perrow's. He wrote a book
called "Normal Accidents." Perrow's Theory of Normal Accidents is that when
independent failures become connected, the probability of catastrophic
failure becomes essentially 100%, given enough time. Again, we can point to
the Gulf and go, "Okay. There are 1600 deep oil rigs running right now.
What are the chances that one of them is going to blow and everybody is
going to stand around scratching their butt going, "what we do?'" The
chances are 100%. It's obvious in hindsight. All this stuff is completely
So, the point here is that we're not doing a good job thinking about, not
hindsight, but we should be thinking with foresight. We could be asking
ourselves when we produce a piece of software, "What happens if this not
only succeeds but what if it succeeds beyond our wildest imaginings?" When
I asked Tom Van Vleck who wrote FTP, one of the things he said I thought
was fantastic. "I'm sure that if whoever it was who wrote it knew it was
going to be so successful, he would've done it differently." Well, yes,
that is in fact the point.
Our software is evolving. We are into an evolutionary lifecycle of
software. The whole point of that extremely long rambling discussion about
what has happened with HTTP is not that we took a look at HTTP and said,
"It needs to be different. Let's fix it." Instead of fixing it, we actually
went interested more stupid stuff to make the stupid stop hurting and we've
been doing more and more stupid stuff for a very long time.
I don't know how much you know about neurology but our eyes see everything
backwards. And our optical systems, because they're actually evolved from
our cerebral cortex, the blood supply for your optic nerves and stuff, your
sensors, are actually on the outside of the sensors. So, if you get this
right, the light, as I'm looking at you, the light from you is going
through the blood supply and then falling on the sensors. If you're a
designer of telescopes, you would never have thought to build something
this way. The reason that our eyes do that is because we had billions of
years to make them not suck as bad as they must've really been when they
first were little spots on the sides of some creatures had been crawling
around someplace. But, we shouldn't design our software that way.
We don't have to design our software this way and, more importantly, we
don't owe our software a sinecure. When we write a piece of bad software,
it's not like your dog. When your dog starts to get old and feeble you try
to make its life comfortable and then you finally put it to sleep as
painlessly as possible. We don't owe our software anything. We could've
taken FTP out and shot it a long time ago and no one would've noticed.
Right? In fact, if you shot FTP around 1976, we probably never even
would've had a firewall industry. We would've had routers that did all this
stuff because it would've been easy for them to do it.
So, some sort of intelligent design would be nice. When I start daydreaming
about strategic software and when people talk to me about the Internet
being the next sort of electronic battle front, the landscape is too stupid
to be worth fighting over. This is just not worth fighting over. If we
actually had someone who actually designed the Internet, it would actually
be much more useful and it would actually be much more valuable and then we
would actually have a serious problem to think about.
So, when you take laziness and genius and you combine it with momentum,
what you're going to get is unpredictable baroqueness and that's what we're
getting with the Internet right now. Why is this mattering? The reason this
matters is because software is now becoming a major cost driver in almost
everything that we build. It used to be that the software costs on the F-18,
the first generation of the F-18 flew in 1986 or something like that.
The software costs for it were about $1 billion. It's up to something like
$18 billion and it's flying with about 18 or 19 million lines of code in
it, which is pretty amazing. It's really, really convoluted stuff.
By the way, just so you guys can all get a good chuckle over this, you can
probably guess how that happened. Right? What happened was that each of the
major weapons systems that display stuff through the HUD of the F-18 was
coded by somebody different. It was a different program. They're treated as
different programs. So, what you basically got is you got this particular
bomb that has its own million lines of code stack. This particular doodad
has its own million lines of code stack. Let's link it all together and fly
it. Keeping that working is why it's getting more and more expensive. We're
at the point now where the program costs, the software to keep the joint
strike fighter from falling out of the sky, costs as much as designing the
airframe. So, we need to be a lot more intelligent about this and I think
that was my point.
So, thank you.