Back to posts

What happens when I run netstat?


posted on:

I wanted to do a deep dive into each step of a common command I use, and I use netstat quite a bit. I will attempt to break down each step into syscalls as detailed by strace.

So what happens when I type netstat -ntpl?

To figure this out I’m going to run strace -f netstat -ntpl

Strace will attach to the process, and show me all of the system calls that are being run under the hood, when we run that target command. The -f tells strace we want to follow that process.

OK. I’ll break this into sections… but here goes

execve

6208  execve("/usr/bin/netstat", ["netstat", "-ntpl"], [/* 55 vars */]) = 0

execve is the call used to execute the program, in this case, our netstat command. execve replaces the current program text of the process, with the new program, netstat.

mmap and brk

6208  brk(NULL)                         = 0x55e2d6044000
6208  mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa3075e1000

We see this a lot, I’m only going to ack it once, here. brk and mmap are used to obtain memory for the task at hand. brk is called to change the location of the program break, which is the end of the data segment for that process. This has the effect of allocation memory to the process. Here we get a chunk of memory once, and then mmap creates new mappings in the process space.

A good quick read on quora about that: How do the brk and mmap system calls work in the Linux kernel?

Loading libraries

6208  access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
6208  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
6208  fstat(3, {st_mode=S_IFREG|0644, st_size=99804, ...}) = 0
6208  mmap(NULL, 99804, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa3075c8000
6208  close(3)                          = 0
  1. access() - This call is just checking to make sure permissions allow the calling process to read the file. R_OK

  2. open() - This actually opens the file, read-only, and uses the close-on-exec flag, that this file handle would be closed were a exec happen before it was explicitly closed.

    We see the file descriptor number 3 returned, which just happens to be the next available descriptor for this process.

  3. fstat() - is used to get information from the open file descriptor (3) into a stat struct. In this case, it’s used to get the size of a memory that mmap will need to map.

  4. mmap() - as seen below, mmap takes a few args

    • *addr - pointer to the address space where we will add this mapping to. Since we have not mapped anything yet, NULL is given.
    • size - which we got from fstat()
    • prot - memory protection, PROT_READ means pages may be read.
    • flags - MAP_PRIVATE creates a copy-on-write mapping, updates are not visible by other processes.
    • fd - file descriptor we are reading into memory
    • offset - this is where in the file we start mapping the n bytes in size
    void *mmap(void *addr, size_t length, int prot, int flags,
            int fd, off_t offset);
    
  5. close() - Now that the file is mapped to memory, it can be safely closed.

mprotect

6208  open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
6208  read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 c\0\0\0\0\0\0"..., 832) = 832
6208  fstat(3, {st_mode=S_IFREG|0755, st_size=153184, ...}) = 0
6208  mmap(NULL, 2253688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fa307197000
6208  mprotect(0x7fa3071ba000, 2097152, PROT_NONE) = 0
6208  mmap(0x7fa3073ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7fa3073ba000
6208  mmap(0x7fa3073bc000, 4984, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fa3073bc000
6208  close(3)= 0

Here you can see a similar section as above, except we have the mprotect call showing up. This call is used to change permissions on a block of memory. But.. we used the bitmask in mmap to do that, why would we need to do it right after? And whats up with different sizes?

Well, turns out this due to the way that ld.so loads the libraries. Without getting too detailed, there are “holes” in the memory that was mapped, that should have PROTO_NONE set on them, as a security measure. There is a good article that breaks it down here: http://www.cs.stevens.edu/~jschauma/810/elf.html

This explains all the follow up calls to mprotect after we load our shared libraries.

arch_prctl

6208  arch_prctl(ARCH_SET_FS, 0x7fa3075c74c0) = 0

arch_prctl() - in this call the FS segment registers are set on thie x86_64 system.

Thread operations

6208  set_tid_address(0x7fa3075c7790)   = 6208
6208  set_robust_list(0x7fa3075c77a0, 24) = 0

These two calls have to do with threads. set_tid_address based on flags called with clone(2), will set the clear_child_tid attribute to the thread pointer given as an arg.

set_robust_list is used when the kernel should record the head of the list of robust futextes owned by the thread calling it.

Signal functions

6208  rt_sigaction(SIGRTMIN, {sa_handler=0x7fa306741b40, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fa30674d5c0}, NULL, 8) = 0
6208  rt_sigaction(SIGRT_1, {sa_handler=0x7fa306741bd0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7fa30674d5c0}, NULL, 8) = 0
6208  rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0

These are functions related to realtime signals.

  1. we register a handler for SIGRTMIN, and SIGRT_1
  2. we unblock those signals, removing them from the process signal mask

Rlimit - Resources limits

getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0

This does what it says, gets resource limits for this process. This returns a {soft, hard limit} struct. We can see here our stack limits are infinite, or whatever the max we can use.

Proc

This is where we actually start doing things. Netstat needs to get its information from somehwere, and that somewhere is /proc. /proc is a in-memory filesystem that stores basically everything we want to know about the state of our system.

One of the things we can see in /proc, is information about processes, by process id (pid). In there we see a directory called, fd. fd contains symlinks of fd numbers, linked to their file/socket.

Netstat is walking proc, and looking for open sockets, since that’s what we asked it for with -l.

6208  open("/proc/8631/fd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
6208  fstat(4, {st_mode=S_IFDIR|0500, st_size=0, ...}) = 0
6208  getdents(4, /* 146 entries */, 32768) = 3504
6208  readlink("/proc/8631/fd/0", "/dev/null", 29) = 9
6208  readlink("/proc/8631/fd/1", "/dev/null", 29) = 9
6208  readlink("/proc/8631/fd/2", "socket:[36353]", 29) = 14
6208  open("/proc/8631/cmdline", O_RDONLY) = 5
6208  read(5, "/home/djohansen/.dropbox-dist/dr"..., 511) = 65
6208  readlink("/proc/8631/fd/85", "/home/djohansen/.dropbox/inst", 29) = 29
...
6208  readlink("/proc/8631/fd/109", "/home/djohansen/.dropbox/inst", 29) = 29
6208  readlink("/proc/8631/fd/110", "socket:[1170940365]", 29) = 19,
6208  open("/proc/8631/attr/current", O_RDONLY|O_CLOEXEC) = 5
6208  read(5, "unconfined_u:unconfined_r:unconf"..., 4095) = 54
6208  close(5)
...

The first thing that happens in this block, is the directory /proc/8631/fd is opened as a directory stream. fstat is called on fd 4, that dir stream, and getdents (get directory entries) is called to get the list of files/dirs in that directory.

We can see readlink is called repeatedly, until we find a socket, at which time the socket information is parsed, triggering a read to current selinux context, which displays the normal open -> fd -> read(fd) -> close(fd) sequence each time.

This goes on until we traverse all pids, all open fds, building a list of listening tcp sockets, with as much info as is granted to the unprivileged user I ran that as.

Write results and exit

6208  write(2, "(Not all processes could be iden"..., 125) = 125
6208  fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 39), ...}) = 0
6208  write(1, "Active Internet connections (onl"..., 43) = 43
6208  write(1, "Proto Recv-Q Send-Q Local Addres"..., 101) = 101
6208  open("/proc/net/tcp", O_RDONLY)   = 3
6208  read(3, "  sl  local_address rem_address "..., 4096) = 4050
6208  write(1, "tcp        0      0 127.0.0.1:63"..., 101) = 101
6208  write(1, "tcp        0      0 127.0.0.1:44"..., 101) = 101
6208  write(1, "tcp        0      0 0.0.0.0:1750"..., 101) = 101
6208  write(1, "tcp        0      0 0.0.0.0:8000"..., 101) = 101
6208  write(1, "tcp        0      0 127.0.0.1:17"..., 101) = 101
6208  write(1, "tcp        0      0 127.0.0.1:17"..., 101) = 101
6208  write(1, "tcp        0      0 0.0.0.0:111 "..., 101) = 101
6208  write(1, "tcp        0      0 192.168.122."..., 101) = 101
6208  read(3, "  26: 0100007F:AD9C 0100007F:BA0"..., 4096) = 4050
6208  read(3, "  53: 0100007F:AD9C 0100007F:BA3"..., 4096) = 4050
6208  read(3, "  80: ED7710AC:9424 7CFD1EC0:01B"..., 4096) = 4050
6208  read(3, " 107: ED7710AC:9C4E 0EA77F26:01B"..., 4096) = 900
6208  read(3, "", 4096)                 = 0
6208  close(3)                          = 0
6208  open("/proc/net/tcp6", O_RDONLY)  = 3
6208  read(3, "  sl  local_address             "..., 4096) = 865
6208  write(1, "tcp6       0      0 ::1:631     "..., 101) = 101
6208  write(1, "tcp6       0      0 :::17500    "..., 101) = 101
6208  write(1, "tcp6       0      0 :::3306     "..., 101) = 101
6208  write(1, "tcp6       0      0 :::111      "..., 101) = 101
6208  read(3, "", 4096)                 = 0
6208  close(3)                          = 0
6208  exit_group(0)                     = ?
6208  +++ exited with 0 +++

The last part of this consists of writing the results to 1,2 fds.

  1. We can see the first write happens to 2, stderr.
  2. fstate to fd1, followed by writes to 1.
  3. /proc/net/tcp is opened and parsed, along with tcp6.
  4. This, combined with the information parsed in proc, is used to write the output to fd 1.
  5. tcp and tcp6 are opened and closed as they are read, and once the last bit has been read/written/closed…
  6. exit_group() is called to terminal all threads in the calling process group..

Conclusion

This was just an overview of system calls used during my one specific call to netstat. There is a ton stuff that happens inside of netstat itself that I didn’t dig into. Hopefully I got most of it though. The source is available at https://sourceforge.net/p/net-tools/code/ci/master/tree/