BPF

BPF之巅学习笔记

第二章

2.3 扩展版BPF

2.4 调用栈回溯

2.5 火焰图

2.6 事件源

2.7 kprobes

内核探测点

2.8 uprobes

用户层探测点

2.9 跟踪点

2.10 USDT

2.11 动态USDT

2.12 性能监控计数器

2.13 perf_events

第三章

3.1 概论

3.2 性能分析方法论

3.2.1 业务负载画像

谁导致了这个负载(eg PID, process name, UID, IP address)?
Why is the load called (code path, stack trace, flame graph)?
What is the load (IOPS, throughput, type)?
How is the load changing over time (per-interval summaries)?

3.2.2 下钻分析

Start examining the highest level.
Examine next-level details.

3.2.3 USE方法论

Utilization 使用率
Saturation 负载率
Errors 错误

3.2.4 检查清单法

3.3 Linux 60s分析

3.3.1 uptime
3.3.2 dmesg|tail
3.3.3 vmstat 1
3.3.4 mpstat -P ALL 1
3.3.5 pidstat 1
3.3.6 iostat -xz 1
3.3.7 free -m
3.3.8 sar -n DEV 1
3.3.9 sar -n TCP,ETCP 1
3.3.10 top

3.4 BCC工具检查清单

3.4.1 execsnoop

统计系统调用exec

1
2
3
4
5
6
# execsnoop
PCOMM PID RET ARGS
supervise 9660 0 ./run
supervise 9661 0 ./run
mkdir 9662 0 /bin/mkdir -p ./main
run 9663 0 ./run
3.4.2 opensnoop

统计系统调用open

1
2
3
4
5
6
7
8
# opensnoop
PID COMM FD ERR PATH
1565 redis-server 5 0 /proc/1565/stat
1603 snmpd 9 0 /proc/net/dev
1603 snmpd 11 0 /proc/net/if_inet6
1603 snmpd -1 2 /sys/class/net/eth0/device/vendor
1603 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms
1603 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time
3.4.3 ext4slower

统计ext4文件系统

1
2
3
4
5
6
7
8
# ext4slower
Tracing ext4 operations slower than 10 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
06:35:01 cron 16464 R 1249 0 16.05 common-auth
06:35:01 cron 16463 R 1249 0 16.04 common-auth
06:35:01 cron 16465 R 1249 0 16.03 common-auth
06:35:01 cron 16465 R 4096 0 10.62 login.defs
06:35:01 cron 16464 R 4096 0 10.61 login.defs
3.4.4 biolatency

bio延迟直方图

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# biolatency -m
Tracing block device I/O... Hit Ctrl-C to end.
^C
msecs : count distribution
0 -> 1 : 16335 |****************************************|
2 -> 3 : 2272 |***** |
4 -> 7 : 3603 |******** |
8 -> 15 : 4328 |********** |
16 -> 31 : 3379 |******** |
32 -> 63 : 5815 |************** |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 1 | |
3.4.5 biosnoop

bio size直方图

1
2
3
4
5
6
7
8
# biosnoop
TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
0.000004001 supervise 1950 xvda1 W 13092560 4096 0.74
0.000178002 supervise 1950 xvda1 W 13092432 4096 0.61
0.001469001 supervise 1956 xvda1 W 13092440 4096 1.24
0.001588002 supervise 1956 xvda1 W 13115128 4096 1.09
1.022346001 supervise 1950 xvda1 W 13115272 4096 0.98
[...]
3.4.6 cachestat

统计文件系统缓存

1
2
3
4
5
6
7
# cachestat
HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
53401 2755 20953 95.09% 14 90223
49599 4098 21460 92.37% 14 90230
16601 2689 61329 86.06% 14 90381
15197 2477 58028 85.99% 14 90522
[...]
3.4.7 tcpconnect

统计系统调用connect

1
2
3
4
5
6
7
8
# tcpconnect
PID COMM IP SADDR DADDR DPORT
1479 telnet 4 127.0.0.1 127.0.0.1 23
1469 curl 4 10.201.219.236 54.245.105.25 80
1469 curl 4 10.201.219.236 54.67.101.145 80
1991 telnet 6 ::1 ::1 23
2015 ssh 6 fe80::2000:bff:fe82:3ac fe80::2000:bff:fe82:3ac 22
[...]
3.4.8 tcpaccept

统计系统调用accept

1
2
3
4
5
6
# tcpaccept
PID COMM IP RADDR LADDR LPORT
907 sshd 4 192.168.56.1 192.168.56.102 22
907 sshd 4 127.0.0.1 127.0.0.1 22
5389 perl 6 1234:ab12:2040:5020:2299:0:5:0 1234:ab12:2040:5020:2299:0:5:0 7001
[...]
3.4.9 tcpretrans

tcp重传统计

1
2
3
4
5
6
# tcpretrans
TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE
01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED
01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED
01:55:17 0 4 10.153.223.157:22 R> 69.53.245.40:22957 ESTABLISHED
[...]
3.4.10 runqlat

cpu上运行队列统计

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# runqlat
Tracing run queue latency... Hit Ctrl-C to end.
^C
usecs : count distribution
0 -> 1 : 233 |*********** |
2 -> 3 : 742 |************************************ |
4 -> 7 : 203 |********** |
8 -> 15 : 173 |******** |
16 -> 31 : 24 |* |
32 -> 63 : 0 | |
64 -> 127 : 30 |* |
128 -> 255 : 6 | |
256 -> 511 : 3 | |
512 -> 1023 : 5 | |
1024 -> 2047 : 27 |* |
2048 -> 4095 : 30 |* |
4096 -> 8191 : 20 | |
8192 -> 16383 : 29 |* |
16384 -> 32767 : 809 |****************************************|
32768 -> 65535 : 64 |*** |
3.4.11 profile

采集CPU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# profile
Sampling at 49 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end.
^C
[...]
copy_user_enhanced_fast_string
copy_user_enhanced_fast_string
_copy_from_iter_full
tcp_sendmsg_locked
tcp_sendmsg
inet_sendmsg
sock_sendmsg
sock_write_iter
new_sync_write
__vfs_write
vfs_write
SyS_write
do_syscall_64
entry_SYSCALL_64_after_hwframe
[unknown]
[unknown]
- iperf (24092)
58

第四章

4.1 BCC的组件
4.2 BCC的特性
4.3 安装BCC
4.4 BCC的工具
4.5 funccount

统计指定函数调用次数

4.6 stackcount

统计栈调用次数

1

4.7 trace

用于跟踪和调试应用程序的执行过程

1

4.8 argdist

函数参数分布统计

4.9 工具文档
4.9.1 Man Page: opensnoop
4.9.2 Examples File: opensnoop
4.10 开发BCC工具
4.11 BCC的内部实现
4.12 BCC的调试
4.12.1 printf() Debugging
4.12.2 BCC Debug output
4.12.3 BCC Debug Flag
4.12.4 bpflist
4.12.5 bpftool
4.12.6 dmesg
4.12.7 Resetting Events

4.13 Summary

第五章

5.1 bpftrace的组件
5.2 bpftrace的特性
5.2.1 bpftrace Event Sources
5.2.2 bpftrace Actions
5.2.3 bpftrace General Features
5.2.4 bpftrace Compared to Other Observability Tools
5.3 bpftrace的安装
5.3.1 Kernel Requirements
5.3.2 Ubuntu
5.3.3 Fedora
5.3.4 Post-Build Steps
5.3.5 Other Distributions
5.4 bpftrace工具
5.4.1 Highlighted Tools
5.4.2 Tool Characteristics
5.4.3 Tool Execution
5.5 bpftrace单行程序
5.6 bpftrace的文档
5.7 bpftrace编程
5.7.1 Usage
5.7.2 Program Structure
5.7.3 Comments
5.7.4 Probe Format
5.7.5 Probe Wildcards
5.7.6 Filters
5.7.7 Actions
5.7.8 Hello, World!
5.7.9 Functions
5.7.10 Variables
5.7.11 Map Functions
5.7.12 Timing vfs_read()
5.8 bpftrace的帮助信息
5.9 bpftrace的探针类型
5.9.1 tracepoint
5.9.2 usdt
5.9.3 kprobe and kretprobe
5.9.4 uprobe and uretprobe
5.9.5 software and hardware
5.9.6 profile and interval
5.10 bpftrace的控制流
5.10.1 Filter
5.10.2 Ternary Operators
5.10.3 If Statements
5.10.4 Unrolled Loops
5.13.3 str()
5.11 bpftrace的运算符
5.12 bpftrace的变量
5.12.1 Built-in Variables
5.12.2 Built-ins: pid, comm, and uid
5.12.3 Built-ins: kstack and ustack
5.12.4 Built-ins: Positional Parameters
5.12.5 Scratch
5.12.6 Maps
5.13 bpftrace的函数
5.13.1 printf()
5.13.2 join()
5.13.3 str()
5.13.4 kstack() and ustack()
5.13.5 ksym() and usym()
5.13.6 kaddr() and uaddr()
5.13.7 system()
5.13.8 exit()
5.14 bpftrace映射表的操作函数
5.14.1 count()
5.14.2 sum(), avg(), min(), and max()
5.14.3 hist()
5.14.4 lhist()
5.14.5 delete()
5.14.6 clear() and zero()
5.14.7 print()
5.15 bpftrace的下一步工作
5.15.1 Explicit Address Modes
5.15.2 Other Additions
5.15.3 ply
5.16 bpftrace的内部运作
5.17 bpftrace的调试
5.17.1 printf() Debugging
5.17.2 Debug Mode
5.17.3 Verbose Mode
5.18 Summary

第六章 CPU

6.1 背景知识
6.1.1 CPU Fundamentals
6.1.2 BPF Capabilities
6.1.3 Strategy
6.2 传统工具
6.2.1 Kernel Statistics
6.2.2 Hardware Statistics
6.2.3 Hardware Sampling
6.2.4 Timed Sampling
6.2.5 Event Statistics and Tracing
6.3 BPF工具
6.3.1 execsnoop
6.3.2 exitsnoop
6.3.3 runqlat
6.3.4 runqlen
6.3.5 runqslower
6.3.6 cpudist
6.3.7 cpufreq
6.3.8 profile
6.3.9 offcputime
6.3.10 syscount
6.3.11 argdist and trace
6.3.12 funccount
6.3.13 softirqs
6.3.14 hardirqs
6.3.15 smpcalls
6.3.16 llcstat
6.3.17 Other Tools
6.4 BPF单行程序
6.4.1 BCC
6.4.2 bpftrace

第七章 内存

7.1 背景知识
7.1.1 Memory Fundamentals
7.1.2 BPF Capabilities
7.1.3 Strategy
7.2 传统工具
7.2.1 Kernel Log
7.2.2 Kernel Statistics
7.2.3 Hardware Statistics and Sampling
7.3 BPF工具
7.3.1 oomkill
7.3.2 memleak
7.3.3 mmapsnoop
7.3.4 brkstack
7.3.5 shmsnoop
7.3.6 faults
7.3.7 ffaults
7.3.8 vmscan
7.3.9 drsnoop
7.3.10 swapin
7.3.11 hfaults
7.3.12 Other Tools
7.4 BPF单行程序
7.4.1 BCC
7.4.2 bpftrace
7.5 Optional Exercises
7.6 Summary

第八章 文件系统

8.1 背景知识
8.1.1 File Systems Fundamentals
8.1.2 BPF Capabilities
8.1.3 Strategy
8.2 传统工具
8.2.1 df
8.2.2 mount
8.2.3 strace
8.2.4 perf
8.2.5 fatrace
8.3 BPF工具
8.3.1 opensnoop
8.3.2 statsnoop
8.3.3 syncsnoop
8.3.4 mmapfiles
8.3.5 scread
8.3.6 fmapfaul
8.3.7 filelife
8.3.8 vfssta
8.3.8 vfssta
8.3.9 vfscount
8.3.10 vfssiz
8.3.11 fsrwsta
8.3.12 fileslower
8.3.13 filetop
8.3.14 writesync
8.3.15 filetype
8.3.16 cachestat
8.3.17 writeback
8.3.18 dcstat
8.3.19 dcsnoop
8.3.20 mountsnoop
8.3.21 xfsslower
8.3.22 xfsdist
8.3.23 ext4dist
8.3.24 icstat
8.3.25 bufgrow
8.3.26 readahead
8.3.27 Other Tools
8.4 BPF单行程序

第九章 磁盘io

9.1 背景知识
9.2 传统工具
9.3 BPF工具
9.3.1 biolatency
9.3.2 biosnoop
9.3.3 biotop
9.3.4 bitesize
9.3.5 seeksize
9.3.6 biopattern
9.3.7 biostacks
9.3.8 bioerr
9.3.9 mdflush
9.3.10 iosched
9.3.11 scsilatency
9.3.12 scsiresult
9.3.13 nvmelatency
9.4 BPF单行程序

第十章 网络

10.1 背景知识
10.2 传统工具
10.3 BPF工具
10.3.1 sockstat
10.3.2 sofamily
10.3.3 soprotocol
10.3.4 soconnect
10.3.5 soaccept
10.3.6 socketio
10.3.7 socksize
10.3.8 sormem
10.3.9 soconnlat
10.3.10 so1stbyte
10.3.11 tcpconnect
10.3.12 tcpaccept
10.3.13 tcplife
10.3.14 tcptop
10.3.15 tcpsnoop
10.3.16 tcpretrans
10.3.17 tcpsynbl
10.3.18 tcpwin
10.3.19 tcpnagle
10.3.20 udpconnect
10.3.21 gethostlatency
10.3.22 ipecn
10.3.23 superping
10.3.24 qdisc-fq
10.3.25 qdisc-cbq, qdisc-cbs, qdisc-codel, qdisc-fq_codel, qdisc-red, and qdisc-tbf
10.3.26 netsize
10.3.27 nettxlat
10.3.28 skbdrop
10.3.29 skblife
10.3.30 ieee80211scan
10.3.31 Other Tools
10.4 BPF单行程序
10.4.1 BCC
10.4.2 bpftrace
10.4.3 BPF One-Liners Examples

第十一章 安全

11.1 背景知识
11.1.1 BPF Capabilities
11.1.2 Unprivileged BPF Users
11.1.3 Configuring BPF Security
11.1.4 Strategy
11.2 BPF工具
11.2.1 execsnoop
11.2.2 elfsnoop
11.2.3 modsnoop
11.2.4 bashreadline
11.2.5 shellsnoop
11.2.6 ttysnoop
11.2.7 opensnoop
11.2.8 eperm
11.2.9 tcpconnect and tcpaccept
11.2.10 tcpreset
11.2.11 capable
11.2.12 setuids
11.3 BPF单行程序
11.3.1 BCC
11.3.2 bpftrace
11.3.3 BPF One-Liners Examples

第十二章 编程语言

12.1 背景知识
12.1.1 Compiled
12.1.2 JIT Compiled
12.1.3 Interpreted
12.1.4 BPF Capabilities
12.1.5 Strategy
12.1.6 BPF Tools
12.2 C
12.2.1 C Function Symbols
12.2.2 C Stack Traces
12.2.3 C Function Tracing
12.2.4 C Function Offset Tracing
12.2.5 C USDT
12.2.6 C One-Liners
12.3 Java
12.3.1 libjvm Tracing
12.3.2 jnistacks
12.3.3 Java Thread Names
12.3.4 Java Method Symbols
12.3.5 Java Stack Traces
12.3.6 Java USDT Probes
12.3.7 profile
12.3.8 offcputime
12.3.9 stackcount
12.3.10 javastat
12.3.11 javathreads
12.3.12 javacalls
12.3.13 javaflow
12.3.14 javagc
12.3.15 javaobjnew
12.3.16 Java One-Liners
12.4 bash shell
12.4.1 Function Counts
12.4.2 Function Argument Tracing (bashfunc.bt)
12.4.3 Function Latency (bashfunclat.bt)
12.4.4 /bin/bash
12.4.5 /bin/bash USDT
12.4.6 bash One-Liners
12.5 其他语言
12.5.1 JavaScript (Node.js)
12.5.2 C++
12.5.3 Golang
12.6 Summary

第十三章 应用程序

13.1 背景知识
13.1.1 Application Fundamentals
13.1.2 Application Example: MySQL Server
13.1.3 BPF Capabilities
13.1.4 Strategy
13.2 BPF工具
13.2.1 execsnoop
13.2.2 threadsnoop
13.2.3 profile
13.2.4 threaded
13.2.5 offcputime
13.2.6 offcpuhist
13.2.7 syscount
13.2.8 ioprofile
13.2.9 libc Frame Pointers
13.2.10 mysqld_qslower
13.2.11 mysqld_clat
13.2.12 signals
13.2.13 killsnoop
13.2.14 pmlock and pmheld
13.2.15 naptime
13.2.16 Other Tools
13.3 单行程序
13.3.1 BCC
13.3.2 bpftrace
13.4 BPF单行程序示例

第十四章 内核

14.1 背景知识
14.1.1 Kernel Fundamentals
14.1.2 BPF Capabilities
14.2 分析策略
14.3 传统工具
14.3.1 Ftrace
14.3.2 perf sched
14.3.3 slabtop
14.3.4 Other Tools
14.4 BPF工具
14.4.1 loads
14.4.2 offcputime
14.4.3 wakeuptime
14.4.4 offwaketime
14.4.5 mlock and mheld
14.4.6 Spin Locks
14.4.7 kmem
14.4.8 kpages
14.4.9 memleak
14.4.10 slabratetop
14.4.11 numamove
14.4.12 workq
14.4.13 Tasklets
14.4.14 Other Tools
14.5 BPF单行程序
14.5.1 BCC
14.5.2 bpftrace
14.6 BPF单行程序示例
14.6.1 Counting System Calls by Syscall Function
14.6.2 Counting hrtimer Starts by Kernel Function
14.7 Challenges
14.8 Summary

第十五章 容器

15.1 背景知识
15.1.1 BPF Capabilities
15.1.2 Challenges
15.1.3 Strategy
15.2 传统工具
15.2.1 From the Host
15.2.2 From the Container
15.2.3 systemd-cgtop
15.2.4 kubectl top
15.2.5 docker stats
15.2.6 /sys/fs/cgroups
15.2.7 perf
15.3 BPF工具
15.3.1 runqlat
15.3.2 pidnss
15.3.3 blkthrot
15.3.4 overlayfs
15.4 BPF单行程序
15.5 可选练习

第十六章 虚拟机管理器

16.1 背景知识
16.1.1 BPF Capabilities
16.1.2 Suggested Strategies
16.2 传统工具
16.3 访客系统的BPF工具
16.3.1 Xen Hypercalls
16.3.2 xenhyper
16.3.3 Xen Callbacks
16.3.4 cpustolen
16.3.5 HVM Exit Tracing
16.4 宿主机的BPF工具
16.4.1 kvmexits
16.4.2 Future Work

第十七章 其他BPF性能工具

17.1 Vector and Performance Co-Pilot (PCP)
17.1.1 Visualizations
17.1.2 Visualization: Heat Maps
17.1.3 Visualization: Tabular Data
17.1.4 BCC Provided Metrics
17.1.5 Internals
17.1.6 Installing PCP and Vector
17.1.7 Connecting and Viewing Data
17.1.8 Configuring the BCC PMDA
17.1.9 Future Work
17.1.10 Further Reading
17.2 Grafana and Performance Co-Pilot (PCP)
17.2.1 Installation and Configuration
17.2.2 Connecting and Viewing Data
17.2.3 Future Work
17.2.4 Further Reading
17.3 Cloudflare eBPF Prometheus Exporter (with Grafana)
17.3.1 Build and Run the ebpf Exporter
17.3.2 Configure Prometheus to Monitor the ebpf_exporter Instance
17.3.3 Set Up a Query in Grafana
17.3.4 Further Reading
17.4 kubectl-trace
17.4.1 Tracing Nodes
17.4.2 Tracing Pods and Containers
17.4.3 Further Reading
17.5 Other Tools
17.6 Summary
Nginx

Nginx的mirror模块详解

1.相关指令

1
2
3
4
5
6
7
8
9
10
Syntax:	mirror uri | off;
Default:	
mirror off;
Context:	http, server, location

Syntax:	mirror_request_body on | off;
Default:	
mirror_request_body on;
Context:	http, server, location

2.配置解析函数

首先看下指令mirror的配置解析函数ngx_http_mirror的实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
static char *
ngx_http_mirror(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_mirror_loc_conf_t *mlcf = conf;

    ngx_str_t  *value, *s;

    value = cf->args->elts;
    //判断参数是否为off
    if (ngx_strcmp(value[1].data, "off") == 0) {
        if (mlcf->mirror != NGX_CONF_UNSET_PTR) {
            return "is duplicate";
        }

        mlcf->mirror = NULL;
        return NGX_CONF_OK;
    }

    if (mlcf->mirror == NULL) {
        return "is duplicate";
    }
    // 构造数组mlcf->mirror
    if (mlcf->mirror == NGX_CONF_UNSET_PTR) {
        mlcf->mirror = ngx_array_create(cf->pool, 4, sizeof(ngx_str_t));
        if (mlcf->mirror == NULL) {
            return NGX_CONF_ERROR;
        }
    }
    // push一个数组元素，用于保存mirror配置
    s = ngx_array_push(mlcf->mirror);
    if (s == NULL) {
        return NGX_CONF_ERROR;
    }
    // 保存参数，也就是url
    *s = value[1];

    return NGX_CONF_OK;
}

接下来看看handler函数ngx_http_mirror_handler的实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
static ngx_int_t
ngx_http_mirror_handler(ngx_http_request_t *r)
{
    ngx_int_t                    rc;
    ngx_http_mirror_ctx_t       *ctx;
    ngx_http_mirror_loc_conf_t  *mlcf;
    // 子请求，直接返回
    if (r != r->main) {
        return NGX_DECLINED;
    }
    // 获取location级别的配置
    mlcf = ngx_http_get_module_loc_conf(r, ngx_http_mirror_module);
    // 没有配置mirror，直接返回
    if (mlcf->mirror == NULL) {
        return NGX_DECLINED;
    }

    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "mirror handler");
    // 处理请求body
    if (mlcf->request_body) {
        ctx = ngx_http_get_module_ctx(r, ngx_http_mirror_module);

        if (ctx) {
            return ctx->status;
        }
        // 申请ctx内存
        ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_mirror_ctx_t));
        if (ctx == NULL) {
            return NGX_ERROR;
        }

        ctx->status = NGX_DONE;
        // 设置ctx
        ngx_http_set_ctx(r, ctx, ngx_http_mirror_module);
        // 读request_body
        rc = ngx_http_read_client_request_body(r, ngx_http_mirror_body_handler);
        if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
            return rc;
        }
        // 结束请求
        ngx_http_finalize_request(r, NGX_DONE);
        return NGX_DONE;
    }
    // 调用handler
    return ngx_http_mirror_handler_internal(r);
}

接下来看看ngx_http_mirror_body_handler处理逻辑

1
2
3
4
5
6
7
8
9
10
11
12
13
14
static void
ngx_http_mirror_body_handler(ngx_http_request_t *r)
{
    ngx_http_mirror_ctx_t  *ctx;
    // get ctx
    ctx = ngx_http_get_module_ctx(r, ngx_http_mirror_module);
    //调用internal
    ctx->status = ngx_http_mirror_handler_internal(r);

    r->preserve_body = 1;
    // 重新执行各个阶段
    r->write_event_handler = ngx_http_core_run_phases;
    ngx_http_core_run_phases(r);
}

接下来看下ngx_http_mirror_init的处理逻辑

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
static ngx_int_t
ngx_http_mirror_init(ngx_conf_t *cf)
{
    ngx_http_handler_pt        *h;
    ngx_http_core_main_conf_t  *cmcf;

    cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
    // precontent阶段，push出一个元素，用于保存handler
    h = ngx_array_push(&cmcf->phases[NGX_HTTP_PRECONTENT_PHASE].handlers);
    if (h == NULL) {
        return NGX_ERROR;
    }
    // handler赋值
    *h = ngx_http_mirror_handler;

    return NGX_OK;
}

接下来看看ngx_http_mirror_handler_internal实现，主要是遍历所有mirror指令，创建子请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
static ngx_int_t
ngx_http_mirror_handler_internal(ngx_http_request_t *r)
{
    ngx_str_t                   *name;
    ngx_uint_t                   i;
    ngx_http_request_t          *sr;
    ngx_http_mirror_loc_conf_t  *mlcf;

    mlcf = ngx_http_get_module_loc_conf(r, ngx_http_mirror_module);

    name = mlcf->mirror->elts;
    // 遍历location下的mirror指令
    for (i = 0; i < mlcf->mirror->nelts; i++) {
        //创建子请求
        if (ngx_http_subrequest(r, &name[i], &r->args, &sr, NULL,
                                NGX_HTTP_SUBREQUEST_BACKGROUND)
            != NGX_OK)
        {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }

        sr->header_only = 1;
        sr->method = r->method;
        sr->method_name = r->method_name;
    }

    return NGX_DECLINED;
}

以上就是mirror模块主要功能

Nginx

Nginx的auth_basic模块详解

1.相关指令

1
2
3
4
5
6
7
8
9
Syntax:	auth_basic string | off;
Default:	
auth_basic off;
Context:	http, server, location, limit_except

Syntax:	auth_basic_user_file file;
Default:	—
Context:	http, server, location, limit_except

2.配置解析函数

首先看下指令auth_basic的配置解析函数ngx_http_set_complex_value_slot，这是个通用的函数，这里不再赘述
接下来看下指令auth_basic_user_file的配置解析函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
static char *
ngx_http_auth_basic_user_file(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    // 模块配置
    ngx_http_auth_basic_loc_conf_t *alcf = conf;

    ngx_str_t                         *value;
    ngx_http_compile_complex_value_t   ccv;

    // 判断是否重复设置
    if (alcf->user_file != NGX_CONF_UNSET_PTR) {
        return "is duplicate";
    }

    //申请内存
    alcf->user_file = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t));
    if (alcf->user_file == NULL) {
        return NGX_CONF_ERROR;
    }

    value = cf->args->elts;
    // 初始化变量
    ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));

    ccv.cf = cf;
    ccv.value = &value[1];
    ccv.complex_value = alcf->user_file;
    ccv.zero = 1;
    ccv.conf_prefix = 1;
    // 编译变量
    if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    return NGX_CONF_OK;
}

3.handler函数

1.接下来看下ngx_http_auth_basic_handler的实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
static ngx_int_t
ngx_http_auth_basic_handler(ngx_http_request_t *r)
{
    off_t                            offset;
    ssize_t                          n;
    ngx_fd_t                         fd;
    ngx_int_t                        rc;
    ngx_err_t                        err;
    ngx_str_t                        pwd, realm, user_file;
    ngx_uint_t                       i, level, login, left, passwd;
    ngx_file_t                       file;
    ngx_http_auth_basic_loc_conf_t  *alcf;
    u_char                           buf[NGX_HTTP_AUTH_BUF_SIZE];
    enum {
        sw_login,
        sw_passwd,
        sw_skip
    } state;

    alcf = ngx_http_get_module_loc_conf(r, ngx_http_auth_basic_module);

    if (alcf->realm == NULL || alcf->user_file == NULL) {
        return NGX_DECLINED;
    }

    // 编译变量
    if (ngx_http_complex_value(r, alcf->realm, &realm) != NGX_OK) {
        return NGX_ERROR;
    }

    if (realm.len == 3 && ngx_strncmp(realm.data, "off", 3) == 0) {
        return NGX_DECLINED;
    }
    // 从authorization header中解析出user以及password
    rc = ngx_http_auth_basic_user(r);
    // 没有解析出user password, 添加必要header，返回401
    if (rc == NGX_DECLINED) {

        ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                      "no user/password was provided for basic authentication");

        return ngx_http_auth_basic_set_realm(r, &realm);
    }

    if (rc == NGX_ERROR) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    // 编译变量
    if (ngx_http_complex_value(r, alcf->user_file, &user_file) != NGX_OK) {
        return NGX_ERROR;
    }

    // 打开文件
    fd = ngx_open_file(user_file.data, NGX_FILE_RDONLY, NGX_FILE_OPEN, 0);
    // 打开失败
    if (fd == NGX_INVALID_FILE) {
        err = ngx_errno;

        if (err == NGX_ENOENT) {
            level = NGX_LOG_ERR;
            rc = NGX_HTTP_FORBIDDEN;

        } else {
            level = NGX_LOG_CRIT;
            rc = NGX_HTTP_INTERNAL_SERVER_ERROR;
        }

        ngx_log_error(level, r->connection->log, err,
                      ngx_open_file_n " \"%s\" failed", user_file.data);

        return rc;
    }

    ngx_memzero(&file, sizeof(ngx_file_t));

    file.fd = fd;
    file.name = user_file;
    file.log = r->connection->log;

    state = sw_login;
    passwd = 0;
    login = 0;
    left = 0;
    offset = 0;

    for ( ;; ) {
        i = left;
        // 读文件
        n = ngx_read_file(&file, buf + left, NGX_HTTP_AUTH_BUF_SIZE - left,
                          offset);

        if (n == NGX_ERROR) {
            rc = NGX_HTTP_INTERNAL_SERVER_ERROR;
            goto cleanup;
        }
        // 文件读完
        if (n == 0) {
            break;
        }
        // 状态机处理
        for (i = left; i < left + n; i++) {
            switch (state) {

            case sw_login:
                if (login == 0) {
                    // 跳过注释
                    if (buf[i] == '#' || buf[i] == CR) {
                        state = sw_skip;


                        break;
                    }
                    // 跳过换行符
                    if (buf[i] == LF) {
                        break;
                    }
                }
                // 长度不匹配，直接跳过
                if (buf[i] != r->headers_in.user.data[login]) {
                    state = sw_skip;
                    break;
                }
                //username匹配完成
                if (login == r->headers_in.user.len) {
                    state = sw_passwd;
                    passwd = i + 1;
                }

                login++;

                break;
            //开始解析密码
            case sw_passwd:
                if (buf[i] == LF || buf[i] == CR || buf[i] == ':') {
                    buf[i] = '\0';

                    pwd.len = i - passwd;
                    pwd.data = &buf[passwd];
                    //处理密码
                    rc = ngx_http_auth_basic_crypt_handler(r, &pwd, &realm);
                    goto cleanup;
                }

                break;

            case sw_skip:
                if (buf[i] == LF) {
                    state = sw_login;
                    login = 0;
                }

                break;
            }
        }
        //密码没有解析完，继续解析
        if (state == sw_passwd) {
            left = left + n - passwd;
            ngx_memmove(buf, &buf[passwd], left);
            passwd = 0;

        } else {
            left = 0;
        }
        // 偏移n
        offset += n;
    }
    // 解析到密码段
    if (state == sw_passwd) {
        pwd.len = i - passwd;
        pwd.data = ngx_pnalloc(r->pool, pwd.len + 1);
        if (pwd.data == NULL) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }
        //拷贝密码
        ngx_cpystrn(pwd.data, &buf[passwd], pwd.len + 1);
        //对密码进行处理
        rc = ngx_http_auth_basic_crypt_handler(r, &pwd, &realm);
        goto cleanup;
    }

    ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                  "user \"%V\" was not found in \"%s\"",
                  &r->headers_in.user, user_file.data);
    // 调用函数，设置auth header
    rc = ngx_http_auth_basic_set_realm(r, &realm);

cleanup:
    // 关闭文件
    if (ngx_close_file(file.fd) == NGX_FILE_ERROR) {
        ngx_log_error(NGX_LOG_ALERT, r->connection->log, ngx_errno,
                      ngx_close_file_n " \"%s\" failed", user_file.data);
    }

    ngx_explicit_memzero(buf, NGX_HTTP_AUTH_BUF_SIZE);

    return rc;
}

接下来看下ngx_http_auth_basic_crypt_handler函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
static ngx_int_t
ngx_http_auth_basic_crypt_handler(ngx_http_request_t *r, ngx_str_t *passwd,
    ngx_str_t *realm)
{
    ngx_int_t   rc;
    u_char     *encrypted;
    // 使用header中的passwd作为key进行编码
    rc = ngx_crypt(r->pool, r->headers_in.passwd.data, passwd->data,
                   &encrypted);

    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "rc: %i user: \"%V\" salt: \"%s\"",
                   rc, &r->headers_in.user, passwd->data);

    if (rc != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
    // 比较编码后的字符串
    if (ngx_strcmp(encrypted, passwd->data) == 0) {
        return NGX_OK;
    }

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "encrypted: \"%s\"", encrypted);

    ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                  "user \"%V\": password mismatch",
                  &r->headers_in.user);

    return ngx_http_auth_basic_set_realm(r, realm);
}

接下来看看ngx_http_auth_basic_set_realm的函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
static ngx_int_t
ngx_http_auth_basic_set_realm(ngx_http_request_t *r, ngx_str_t *realm)
{
    size_t   len;
    u_char  *basic, *p;
    // 从headers list中获取一个元素
    r->headers_out.www_authenticate = ngx_list_push(&r->headers_out.headers);
    if (r->headers_out.www_authenticate == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
    //初始化header len
    len = sizeof("Basic realm=\"\"") - 1 + realm->len;
    // 申请内存
    basic = ngx_pnalloc(r->pool, len);
    if (basic == NULL) {
        r->headers_out.www_authenticate->hash = 0;
        r->headers_out.www_authenticate = NULL;
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
    //复制header
    p = ngx_cpymem(basic, "Basic realm=\"", sizeof("Basic realm=\"") - 1);
    p = ngx_cpymem(p, realm->data, realm->len);
    *p = '"';
    // 设置header
    r->headers_out.www_authenticate->hash = 1;
    ngx_str_set(&r->headers_out.www_authenticate->key, "WWW-Authenticate");
    r->headers_out.www_authenticate->value.data = basic;
    r->headers_out.www_authenticate->value.len = len;
    //返回401
    return NGX_HTTP_UNAUTHORIZED;
}

以上就是这个auth-basic模块的实现

Nginx

Nginx的geoip模块详解

1.相关指令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Syntax:	geoip_country file;
Default:	—
Context:	http

Syntax:	geoip_city file;
Default:	—
Context:	http

Syntax:	geoip_org file;
Default:	—
Context:	http
This directive appeared in version 1.0.3.


Syntax:	geoip_proxy address | CIDR;
Default:	—
Context:	http
This directive appeared in versions 1.3.0 and 1.2.1.

Syntax:	geoip_proxy_recursive on | off;
Default:	
geoip_proxy_recursive off;
Context:	http
This directive appeared in versions 1.3.0 and 1.2.1.

2.配置解析函数

首先看下配置解析函数ngx_http_geoip_country

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
static char *
ngx_http_geoip_country(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_geoip_conf_t  *gcf = conf;

    ngx_str_t  *value;
    //重复配置校验
    if (gcf->country) {
        return "is duplicate";
    }
    // 获取指令后面的参数
    value = cf->args->elts;
    // 打开数据文件
    gcf->country = GeoIP_open((char *) value[1].data, GEOIP_MEMORY_CACHE);

    if (gcf->country == NULL) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "GeoIP_open(\"%V\") failed", &value[1]);

        return NGX_CONF_ERROR;
    }
    // 判断参数个数，设置编码格式
    if (cf->args->nelts == 3) {
        if (ngx_strcmp(value[2].data, "utf8") == 0) {
            GeoIP_set_charset(gcf->country, GEOIP_CHARSET_UTF8);

        } else {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid parameter \"%V\"", &value[2]);
            return NGX_CONF_ERROR;
        }
    }

    switch (gcf->country->databaseType) {

    case GEOIP_COUNTRY_EDITION:

        return NGX_CONF_OK;

#if (NGX_HAVE_GEOIP_V6)
    case GEOIP_COUNTRY_EDITION_V6:

        gcf->country_v6 = 1;
        return NGX_CONF_OK;
#endif

    default:
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "invalid GeoIP database \"%V\" type:%d",
                           &value[1], gcf->country->databaseType);
        return NGX_CONF_ERROR;
    }
}

接下来看下配置解析函数ngx_http_geoip_org

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
static char *
ngx_http_geoip_org(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_geoip_conf_t  *gcf = conf;

    ngx_str_t  *value;
    // 重复配置检查
    if (gcf->org) {
        return "is duplicate";
    }

    value = cf->args->elts;
    // 打开数据文件
    gcf->org = GeoIP_open((char *) value[1].data, GEOIP_MEMORY_CACHE);

    if (gcf->org == NULL) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "GeoIP_open(\"%V\") failed", &value[1]);

        return NGX_CONF_ERROR;
    }

    // 设置文件编码格式
    if (cf->args->nelts == 3) {
        if (ngx_strcmp(value[2].data, "utf8") == 0) {
            GeoIP_set_charset(gcf->org, GEOIP_CHARSET_UTF8);

        } else {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid parameter \"%V\"", &value[2]);
            return NGX_CONF_ERROR;
        }
    }
    //选择数据类型
    switch (gcf->org->databaseType) {

    case GEOIP_ISP_EDITION:
    case GEOIP_ORG_EDITION:
    case GEOIP_DOMAIN_EDITION:
    case GEOIP_ASNUM_EDITION:

        return NGX_CONF_OK;

#if (NGX_HAVE_GEOIP_V6)
    case GEOIP_ISP_EDITION_V6:
    case GEOIP_ORG_EDITION_V6:
    case GEOIP_DOMAIN_EDITION_V6:
    case GEOIP_ASNUM_EDITION_V6:

        gcf->org_v6 = 1;
        return NGX_CONF_OK;
#endif

    default:
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "invalid GeoIP database \"%V\" type:%d",
                           &value[1], gcf->org->databaseType);
        return NGX_CONF_ERROR;
    }
}

接下来看下配置解析函数ngx_http_geoip_city

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
static char *
ngx_http_geoip_city(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_geoip_conf_t  *gcf = conf;

    ngx_str_t  *value;
    // 重复设置判断
    if (gcf->city) {
        return "is duplicate";
    }

    value = cf->args->elts;
    // 打开数据文件
    gcf->city = GeoIP_open((char *) value[1].data, GEOIP_MEMORY_CACHE);

    if (gcf->city == NULL) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "GeoIP_open(\"%V\") failed", &value[1]);

        return NGX_CONF_ERROR;
    }
    //处理编码格式
    if (cf->args->nelts == 3) {
        if (ngx_strcmp(value[2].data, "utf8") == 0) {
            GeoIP_set_charset(gcf->city, GEOIP_CHARSET_UTF8);

        } else {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid parameter \"%V\"", &value[2]);
            return NGX_CONF_ERROR;
        }
    }
    // 针对数据类型做处理
    switch (gcf->city->databaseType) {

    case GEOIP_CITY_EDITION_REV0:
    case GEOIP_CITY_EDITION_REV1:

        return NGX_CONF_OK;

#if (NGX_HAVE_GEOIP_V6)
    case GEOIP_CITY_EDITION_REV0_V6:
    case GEOIP_CITY_EDITION_REV1_V6:

        gcf->city_v6 = 1;
        return NGX_CONF_OK;
#endif

    default:
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "invalid GeoIP City database \"%V\" type:%d",
                           &value[1], gcf->city->databaseType);
        return NGX_CONF_ERROR;
    }
}

接下来看下配置解析函数ngx_http_geoip_proxy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
static char *
ngx_http_geoip_proxy(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_geoip_srv_conf_t  *gscf = conf;

    ngx_str_t   *value;
    ngx_cidr_t  cidr, *c;

    value = cf->args->elts;
    // 解析cidr地址
    if (ngx_http_geoip_cidr_value(cf, &value[1], &cidr) != NGX_OK) {
        return NGX_CONF_ERROR;
    }
    // 初始化proxy数组
    if (gscf->proxies == NGX_CONF_UNSET_PTR) {
        gscf->proxies = ngx_array_create(cf->pool, 4, sizeof(ngx_cidr_t));
        if (gscf->proxies == NULL) {
            return NGX_CONF_ERROR;
        }
    }
    // push 一个元素用于保存cidr
    c = ngx_array_push(gscf->proxies);
    if (c == NULL) {
        return NGX_CONF_ERROR;
    }
    // 保存cidr地址
    *c = cidr;

    return NGX_CONF_OK;
}

指令geoip_proxy_recursive使用原生函数ngx_conf_set_flag_slot，这里就不再赘述

3.hander函数

此模块没有handler函数

4.变量处理函数

此模块功能都是为变量实现，所以主要函数都是为变量实现
首先来看下变量geoip_country_code的处理函数ngx_http_geoip_country_variable

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
static ngx_int_t
ngx_http_geoip_country_variable(ngx_http_request_t *r,
    ngx_http_variable_value_t *v, uintptr_t data)
{
    // 根据data类型获取对应函数指针，ngx_http_geoip_country_functions在代码有定义
    ngx_http_geoip_variable_handler_pt     handler =
        ngx_http_geoip_country_functions[data];
#if (NGX_HAVE_GEOIP_V6)
    ngx_http_geoip_variable_handler_v6_pt  handler_v6 =
        ngx_http_geoip_country_v6_functions[data];
#endif

    const char             *val;
    ngx_http_geoip_conf_t  *gcf;
    // 获取main级别conf
    gcf = ngx_http_get_module_main_conf(r, ngx_http_geoip_module);
    // 没有配置country数据
    if (gcf->country == NULL) {
        goto not_found;
    }
    //在handler内部调用ngx_http_geoip_addr
#if (NGX_HAVE_GEOIP_V6)
    val = gcf->country_v6
              ? handler_v6(gcf->country, ngx_http_geoip_addr_v6(r, gcf))
              : handler(gcf->country, ngx_http_geoip_addr(r, gcf));
#else
    val = handler(gcf->country, ngx_http_geoip_addr(r, gcf));
#endif

    if (val == NULL) {
        goto not_found;
    }
    //为变量赋值
    v->len = ngx_strlen(val);
    v->valid = 1;
    v->no_cacheable = 0;
    v->not_found = 0;
    v->data = (u_char *) val;

    return NGX_OK;

not_found:

    v->not_found = 1;

    return NGX_OK;
}

接下来看看ngx_http_geoip_addr函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
static u_long
ngx_http_geoip_addr(ngx_http_request_t *r, ngx_http_geoip_conf_t *gcf)
{
    ngx_addr_t           addr;
    ngx_array_t         *xfwd;
    struct sockaddr_in  *sin;

    addr.sockaddr = r->connection->sockaddr;
    addr.socklen = r->connection->socklen;
    /* addr.name = r->connection->addr_text; */

    xfwd = &r->headers_in.x_forwarded_for;
    //获取server级别配置
    ngx_http_geoip_srv_conf_t  *gscf = ngx_http_get_module_srv_conf(r, ngx_http_geoip_module);
    //从xff头中获取ip
    if (xfwd->nelts > 0 && gscf->proxies != NULL) {
         (void) ngx_http_get_forwarded_addr(r, &addr, xfwd, NULL,
                                           gscf->proxies, gscf->proxy_recursive);
    }

#if (NGX_HAVE_INET6)
    //判断是否为IPV6
    if (addr.sockaddr->sa_family == AF_INET6) {
        u_char           *p;
        in_addr_t         inaddr;
        struct in6_addr  *inaddr6;

        inaddr6 = &((struct sockaddr_in6 *) addr.sockaddr)->sin6_addr;

        if (IN6_IS_ADDR_V4MAPPED(inaddr6)) {
            p = inaddr6->s6_addr;

            inaddr = p[12] << 24;
            inaddr += p[13] << 16;
            inaddr += p[14] << 8;
            inaddr += p[15];

            return inaddr;
        }
    }

#endif
    //非ipv4协议
    if (addr.sockaddr->sa_family != AF_INET) {
        return INADDR_NONE;
    }
    //返回主机字节序
    sin = (struct sockaddr_in *) addr.sockaddr;
    return ntohl(sin->sin_addr.s_addr);
}

接下来看下ngx_http_geoip_city_variable函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
static ngx_int_t
ngx_http_geoip_city_variable(ngx_http_request_t *r,
    ngx_http_variable_value_t *v, uintptr_t data)
{
    char         *val;
    size_t        len;
    GeoIPRecord  *gr;
    //获取city record
    gr = ngx_http_geoip_get_city_record(r);
    if (gr == NULL) {
        goto not_found;
    }

    //获取val的值
    val = *(char **) ((char *) gr + data);
    if (val == NULL) {
        goto no_value;
    }
    //获取变量长度
    len = ngx_strlen(val);
    v->data = ngx_pnalloc(r->pool, len);
    if (v->data == NULL) {
        GeoIPRecord_delete(gr);
        return NGX_ERROR;
    }
    //复制变量值
    ngx_memcpy(v->data, val, len);

    v->len = len;
    v->valid = 1;
    v->no_cacheable = 0;
    v->not_found = 0;
    //回收geoip资源
    GeoIPRecord_delete(gr);

    return NGX_OK;

no_value:

    GeoIPRecord_delete(gr);

not_found:

    v->not_found = 1;

    return NGX_OK;
}

以上就是geoip模块的所有内容

Nginx

Nginx的try_files模块

1.相关指令

1
2
3
4
Syntax:	try_files file ... uri;
try_files file ... =code;
Default:	—
Context:	server, location

2.配置解析函数

首先看下配置解析函数ngx_http_try_files

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
static char *
ngx_http_try_files(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_try_files_loc_conf_t *tlcf = conf;

    ngx_str_t                  *value;
    ngx_int_t                   code;
    ngx_uint_t                  i, n;
    ngx_http_try_file_t        *tf;
    ngx_http_script_compile_t   sc;

    //判断是否重复设置
    if (tlcf->try_files) {
        return "is duplicate";
    }

    // 申请必要内存，根据参数申请内存
    tf = ngx_pcalloc(cf->pool, cf->args->nelts * sizeof(ngx_http_try_file_t));
    if (tf == NULL) {
        return NGX_CONF_ERROR;
    }

    tlcf->try_files = tf;

    value = cf->args->elts;

    // 解析参数
    for (i = 0; i < cf->args->nelts - 1; i++) {

        tf[i].name = value[i + 1];
        // 判断当前参数是否可以作为文件查找的目录，如果是，则将test_dir置位
        if (tf[i].name.len > 0
            && tf[i].name.data[tf[i].name.len - 1] == '/'
            && i + 2 < cf->args->nelts)
        {
            tf[i].test_dir = 1;
            tf[i].name.len--;
            tf[i].name.data[tf[i].name.len] = '\0';
        }

        // 计算参数中的变量的个数
        n = ngx_http_script_variables_count(&tf[i].name);

        // 处理变量
        if (n) {
            ngx_memzero(&sc, sizeof(ngx_http_script_compile_t));

            sc.cf = cf;
            sc.source = &tf[i].name;
            sc.lengths = &tf[i].lengths;
            sc.values = &tf[i].values;
            sc.variables = n;
            sc.complete_lengths = 1;
            sc.complete_values = 1;

            if (ngx_http_script_compile(&sc) != NGX_OK) {
                return NGX_CONF_ERROR;
            }

        } else {
            /* add trailing '\0' to length */
            tf[i].name.len++;
        }
    }
    //如果最后一个参数第一个字符是=，则解析最后一个参数的code
    if (tf[i - 1].name.data[0] == '=') {

        code = ngx_atoi(tf[i - 1].name.data + 1, tf[i - 1].name.len - 2);

        if (code == NGX_ERROR || code > 999) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid code \"%*s\"",
                               tf[i - 1].name.len - 1, tf[i - 1].name.data);
            return NGX_CONF_ERROR;
        }

        tf[i].code = code;
    }

    return NGX_CONF_OK;
}

filter模块需要在配置解析完成之后，在适当的阶段介入，在ngx_http_try_files_module_ctx中设置

1
2
3
4
5
6
7
8
9
10
11
12
13
static ngx_http_module_t  ngx_http_try_files_module_ctx = {
    NULL,                                  /* preconfiguration */
    ngx_http_try_files_init,               /* postconfiguration */

    NULL,                                  /* create main configuration */
    NULL,                                  /* init main configuration */

    NULL,                                  /* create server configuration */
    NULL,                                  /* merge server configuration */

    ngx_http_try_files_create_loc_conf,    /* create location configuration */
    NULL                                   /* merge location configuration */
};

看下ngx_http_try_files_init函数实现，可以看到try_files模块的filter在precontent阶段介入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
static ngx_int_t
ngx_http_try_files_init(ngx_conf_t *cf)
{
    ngx_http_handler_pt        *h;
    ngx_http_core_main_conf_t  *cmcf;

    cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);

    // 从precontent阶段的handler中取一个
    h = ngx_array_push(&cmcf->phases[NGX_HTTP_PRECONTENT_PHASE].handlers);
    if (h == NULL) {
        return NGX_ERROR;
    }

    *h = ngx_http_try_files_handler;

    return NGX_OK;
}

接下来看下ngx_http_try_files_handler模块具体实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
static ngx_int_t
ngx_http_try_files_handler(ngx_http_request_t *r)
{
    size_t                          len, root, alias, reserve, allocated;
    u_char                         *p, *name;
    ngx_str_t                       path, args;
    ngx_uint_t                      test_dir;
    ngx_http_try_file_t            *tf;
    ngx_open_file_info_t            of;
    ngx_http_script_code_pt         code;
    ngx_http_script_engine_t        e;
    ngx_http_core_loc_conf_t       *clcf;
    ngx_http_script_len_code_pt     lcode;
    ngx_http_try_files_loc_conf_t  *tlcf;

    tlcf = ngx_http_get_module_loc_conf(r, ngx_http_try_files_module);
    //如果没有配置try_file直接返回
    if (tlcf->try_files == NULL) {
        return NGX_DECLINED;
    }

    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "try files handler");

    allocated = 0;
    root = 0;
    name = NULL;
    /* suppress MSVC warning */
    path.data = NULL;

    tf = tlcf->try_files;

    clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
    // 获取location下配置的alias
    alias = clcf->alias;

    for ( ;; ) {
        // 解析try_files后面的变量
        if (tf->lengths) {
            ngx_memzero(&e, sizeof(ngx_http_script_engine_t));

            e.ip = tf->lengths->elts;
            e.request = r;

            /* 1 is for terminating '\0' as in static names */
            len = 1;

            while (*(uintptr_t *) e.ip) {
                lcode = *(ngx_http_script_len_code_pt *) e.ip;
                len += lcode(&e);
            }

        } else {
            len = tf->name.len;
        }
        // 如果没有alias，则判断try_files第一个参数长度与uri的长度
        if (!alias) {
            reserve = len > r->uri.len ? len - r->uri.len : 0;

        } else if (alias == NGX_MAX_SIZE_T_VALUE) {
            reserve = len;

        } else {
            reserve = len > r->uri.len - alias ? len - (r->uri.len - alias) : 0;
        }

        if (reserve > allocated || !allocated) {

            /* 16 bytes are preallocation */
            allocated = reserve + 16;
            // 根据uri映射磁盘文件
            if (ngx_http_map_uri_to_path(r, &path, &root, allocated) == NULL) {
                return NGX_HTTP_INTERNAL_SERVER_ERROR;
            }

            name = path.data + root;
        }

        if (tf->values == NULL) {

            /* tf->name.len includes the terminating '\0' */

            ngx_memcpy(name, tf->name.data, tf->name.len);

            path.len = (name + tf->name.len - 1) - path.data;

        } else {
            e.ip = tf->values->elts;
            e.pos = name;
            e.flushed = 1;
            //处理变量
            while (*(uintptr_t *) e.ip) {
                code = *(ngx_http_script_code_pt *) e.ip;
                code((ngx_http_script_engine_t *) &e);
            }

            path.len = e.pos - path.data;

            *e.pos = '\0';

            if (alias && alias != NGX_MAX_SIZE_T_VALUE
                && ngx_strncmp(name, r->uri.data, alias) == 0)
            {
                ngx_memmove(name, name + alias, len - alias);
                path.len -= alias;
            }
        }

        test_dir = tf->test_dir;
        // 指针后移
        tf++;

        ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "trying to use %s: \"%s\" \"%s\"",
                       test_dir ? "dir" : "file", name, path.data);

        if (tf->lengths == NULL && tf->name.len == 0) {
            // 如果最后一个参数code有设置，则直接返回该code
            if (tf->code) {
                return tf->code;
            }

            path.len -= root;
            path.data += root;
            // 如果是@开头的path，则跳转到对应location
            if (path.data[0] == '@') {
                (void) ngx_http_named_location(r, &path);

            } else {
                ngx_http_split_args(r, &path, &args);
                //内部重定向
                (void) ngx_http_internal_redirect(r, &path, &args);
            }
            // 结束请求
            ngx_http_finalize_request(r, NGX_DONE);
            return NGX_DONE;
        }

        ngx_memzero(&of, sizeof(ngx_open_file_info_t));
        // 设置cache_file参数
        of.read_ahead = clcf->read_ahead;
        of.directio = clcf->directio;
        of.valid = clcf->open_file_cache_valid;
        of.min_uses = clcf->open_file_cache_min_uses;
        of.test_only = 1;
        of.errors = clcf->open_file_cache_errors;
        of.events = clcf->open_file_cache_events;

        // 处理软链接
        if (ngx_http_set_disable_symlinks(r, clcf, &path, &of) != NGX_OK) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }
        // 打开文件
        if (ngx_open_cached_file(clcf->open_file_cache, &path, &of, r->pool)
            != NGX_OK)
        {
            if (of.err == 0) {
                return NGX_HTTP_INTERNAL_SERVER_ERROR;
            }

            if (of.err != NGX_ENOENT
                && of.err != NGX_ENOTDIR
                && of.err != NGX_ENAMETOOLONG)
            {
                ngx_log_error(NGX_LOG_CRIT, r->connection->log, of.err,
                              "%s \"%s\" failed", of.failed, path.data);
            }

            continue;
        }
        // 当前目录不可用于查找文件
        if (of.is_dir != test_dir) {
            continue;
        }

        path.len -= root;
        path.data += root;

        if (!alias) {
            r->uri = path;

        } else if (alias == NGX_MAX_SIZE_T_VALUE) {
            if (!test_dir) {
                r->uri = path;
                r->add_uri_to_alias = 1;
            }

        } else {
            name = r->uri.data;

            r->uri.len = alias + path.len;
            r->uri.data = ngx_pnalloc(r->pool, r->uri.len);
            if (r->uri.data == NULL) {
                r->uri.len = 0;
                return NGX_HTTP_INTERNAL_SERVER_ERROR;
            }

            p = ngx_copy(r->uri.data, name, alias);
            // 保存path
            ngx_memcpy(p, path.data, path.len);
        }
        // 设置文件后缀
        ngx_http_set_exten(r);

        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "try file uri: \"%V\"", &r->uri);

        return NGX_DECLINED;
    }

    /* not reached */

3.总结

以上就是try_files模块的详解，从指令到handler都进行了一些说明。

Nginx

Nginx的header_filter模块

1.相关指令

header_filter模块主要有以下几个指令

1
2
3
Syntax:	add_header name value [always];
Default:	—

1
2
3
4
5
Context:	http, server, location, if in location
Syntax:	add_trailer name value [always];
Default:	—
Context:	http, server, location, if in location
This directive appeared in version 1.13.2.
1
2
3
4
5
6
Syntax:	expires [modified] time;
expires epoch | max | off;
Default:	
expires off;
Context:	http, server, location, if in location

配置示例

1
2
3
4
5
6
7
8
9
expires    24h;
expires    modified +24h;
expires    @24h;
expires    0;
expires    -1;
expires    epoch;
expires    $expires;
add_header Cache-Control private;

2. 源码解析

2.1 配置解析函数

  1. 首先来看下add_header指令解析函数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
static char *
ngx_http_headers_add(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_headers_conf_t *hcf = conf;

    ngx_str_t                          *value;
    ngx_uint_t                          i;
    ngx_array_t                       **headers;
    ngx_http_header_val_t              *hv;
    ngx_http_set_header_t              *set;
    ngx_http_compile_complex_value_t    ccv;

    value = cf->args->elts;

    // 取到数组指针
    headers = (ngx_array_t **) ((char *) hcf + cmd->offset);

    if (*headers == NULL) {
        *headers = ngx_array_create(cf->pool, 1,
                                    sizeof(ngx_http_header_val_t));
        if (*headers == NULL) {
            return NGX_CONF_ERROR;
        }
    }

    //从数组中push一个元素
    hv = ngx_array_push(*headers);
    if (hv == NULL) {
        return NGX_CONF_ERROR;
    }

    // 取配置header的key
    hv->key = value[1];
    hv->handler = NULL;
    hv->offset = 0;
    hv->always = 0;

    if (headers == &hcf->headers) {
        hv->handler = ngx_http_add_header;

        set = ngx_http_set_headers;
        for (i = 0; set[i].name.len; i++) {
            if (ngx_strcasecmp(value[1].data, set[i].name.data) != 0) {
                continue;
            }

            hv->offset = set[i].offset;
            hv->handler = set[i].handler;

            break;
        }
    }

    if (value[2].len == 0) {
        ngx_memzero(&hv->value, sizeof(ngx_http_complex_value_t));

    } else {
        ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));

        ccv.cf = cf;
        ccv.value = &value[2];
        ccv.complex_value = &hv->value;

        if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }

    if (cf->args->nelts == 3) {
        return NGX_CONF_OK;
    }

    if (ngx_strcmp(value[3].data, "always") != 0) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "invalid parameter \"%V\"", &value[3]);
        return NGX_CONF_ERROR;
    }

    hv->always = 1;

    return NGX_CONF_OK;
}

  1. 接下来看下expires指令解析函数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
static char *
ngx_http_headers_expires(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_headers_conf_t *hcf = conf;

    char                              *err;
    ngx_str_t                         *value;
    ngx_int_t                          rc;
    ngx_uint_t                         n;
    ngx_http_complex_value_t           cv;
    ngx_http_compile_complex_value_t   ccv;

    // 判断是否已经设置过该指令
    if (hcf->expires != NGX_HTTP_EXPIRES_UNSET) {
        return "is duplicate";
    }

    value = cf->args->elts;

    //判断指令后的参数
    if (cf->args->nelts == 2) {

        hcf->expires = NGX_HTTP_EXPIRES_ACCESS;

        n = 1;

    } else { /* cf->args->nelts == 3 */

        if (ngx_strcmp(value[1].data, "modified") != 0) {
            return "invalid value";
        }

        hcf->expires = NGX_HTTP_EXPIRES_MODIFIED;

        n = 2;
    }

    // 准备变量
    ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));

    ccv.cf = cf;
    ccv.value = &value[n];
    ccv.complex_value = &cv;

    //编译变量
    if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    if (cv.lengths != NULL) {

        hcf->expires_value = ngx_palloc(cf->pool,
                                        sizeof(ngx_http_complex_value_t));
        if (hcf->expires_value == NULL) {
            return NGX_CONF_ERROR;
        }

        *hcf->expires_value = cv;

        return NGX_CONF_OK;
    }

    rc = ngx_http_parse_expires(&value[n], &hcf->expires, &hcf->expires_time,
                                &err);
    if (rc != NGX_OK) {
        return err;
    }

    return NGX_CONF_OK;
}

2.2 filter函数

2.2.1 headers_filter函数

首先看下ngx_http_headers_filter函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
static ngx_int_t
ngx_http_headers_filter(ngx_http_request_t *r)
{
    ngx_str_t                 value;
    ngx_uint_t                i, safe_status;
    ngx_http_header_val_t    *h;
    ngx_http_headers_conf_t  *conf;

    // 子请求不做处理，直接返回
    if (r != r->main) {
        return ngx_http_next_header_filter(r);
    }

    // 获取location级别配置
    conf = ngx_http_get_module_loc_conf(r, ngx_http_headers_filter_module);

    if (conf->expires == NGX_HTTP_EXPIRES_OFF
        && conf->headers == NULL
        && conf->trailers == NULL)
    {
        return ngx_http_next_header_filter(r);
    }

    switch (r->headers_out.status) {

    case NGX_HTTP_OK:
    case NGX_HTTP_CREATED:
    case NGX_HTTP_NO_CONTENT:
    case NGX_HTTP_PARTIAL_CONTENT:
    case NGX_HTTP_MOVED_PERMANENTLY:
    case NGX_HTTP_MOVED_TEMPORARILY:
    case NGX_HTTP_SEE_OTHER:
    case NGX_HTTP_NOT_MODIFIED:
    case NGX_HTTP_TEMPORARY_REDIRECT:
    case NGX_HTTP_PERMANENT_REDIRECT:
        safe_status = 1;
        break;

    default:
        safe_status = 0;
        break;
    }

    if (conf->expires != NGX_HTTP_EXPIRES_OFF && safe_status) {
        if (ngx_http_set_expires(r, conf) != NGX_OK) {
            return NGX_ERROR;
        }
    }

    if (conf->headers) {
        h = conf->headers->elts;
        for (i = 0; i < conf->headers->nelts; i++) {

            if (!safe_status && !h[i].always) {
                continue;
            }

            if (ngx_http_complex_value(r, &h[i].value, &value) != NGX_OK) {
                return NGX_ERROR;
            }

            if (h[i].handler(r, &h[i], &value) != NGX_OK) {
                return NGX_ERROR;
            }
        }
    }

    if (conf->trailers) {
        h = conf->trailers->elts;
        for (i = 0; i < conf->trailers->nelts; i++) {

            if (!safe_status && !h[i].always) {
                continue;
            }

            r->expect_trailers = 1;
            break;
        }
    }

    return ngx_http_next_header_filter(r);
}

2.2.2 ngx_http_trailers_filter函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
static ngx_int_t
ngx_http_trailers_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
    ngx_str_t                 value;
    ngx_uint_t                i, safe_status;
    ngx_chain_t              *cl;
    ngx_table_elt_t          *t;
    ngx_http_header_val_t    *h;
    ngx_http_headers_conf_t  *conf;

    //获取location级别配置
    conf = ngx_http_get_module_loc_conf(r, ngx_http_headers_filter_module);

    // 如果满足如下条件，则跳过本filter
    if (in == NULL
        || conf->trailers == NULL
        || !r->expect_trailers
        || r->header_only)
    {
        return ngx_http_next_body_filter(r, in);
    }

    // 找到最后一个buf
    for (cl = in; cl; cl = cl->next) {
        if (cl->buf->last_buf) {
            break;
        }
    }

    // 如果cl为null，则跳过本filter
    if (cl == NULL) {
        return ngx_http_next_body_filter(r, in);
    }

    // 根据响应码做选择
    switch (r->headers_out.status) {

    case NGX_HTTP_OK:
    case NGX_HTTP_CREATED:
    case NGX_HTTP_NO_CONTENT:
    case NGX_HTTP_PARTIAL_CONTENT:
    case NGX_HTTP_MOVED_PERMANENTLY:
    case NGX_HTTP_MOVED_TEMPORARILY:
    case NGX_HTTP_SEE_OTHER:
    case NGX_HTTP_NOT_MODIFIED:
    case NGX_HTTP_TEMPORARY_REDIRECT:
    case NGX_HTTP_PERMANENT_REDIRECT:
        safe_status = 1;
        break;

    default:
        safe_status = 0;
        break;
    }

    // 遍历配置的所有trailer
    h = conf->trailers->elts;
    for (i = 0; i < conf->trailers->nelts; i++) {

        if (!safe_status && !h[i].always) {
            continue;
        }

        if (ngx_http_complex_value(r, &h[i].value, &value) != NGX_OK) {
            return NGX_ERROR;
        }

        if (value.len) {
            t = ngx_list_push(&r->headers_out.trailers);
            if (t == NULL) {
                return NGX_ERROR;
            }

            t->key = h[i].key;
            t->value = value;
            t->hash = 1;
        }
    }

    return ngx_http_next_body_filter(r, in);
}

systemtap调试技巧

1、安装

Centos

1.1 必要库

1
yum -y install elfutils、gcc、kernel-devel、kernel-debuginfo、yum-utils

1.2 yum安装

1
yum -y systemtap

Ubuntu

1.1 必要库

1
2
3
4
$ apt-get install build-essential
$ apt-get install gettext
$ apt-get install elfutils
$ apt-get install libdw-dev

1.2 源码安装

1
2
3
4
5
$ git clone git://sourceware.org/git/systemtap.git
$ cd systemtap
$ git checkout release-4.9
$ ./configure
$ make -j (getconf _NPROCESSORS_ONLN)

目前需要使用release4.9分支，安装完成后，在/usr/local/systemtap

2、入门

2.1、简介

SystemTap是一个Linux非常有用的调试（跟踪/探测）工具，常用于Linux内核或者应用程序的信息采集，需要编译时，指定-g选项，比如：获取一个函数里面运行时的变量、调用堆栈，甚至可以直接修改变量的值，对诊断性能或功能问题非常有帮助。SystemTap提供非常简单的命令行接口和很简洁的脚本语言，以及非常丰富的tapset和例子。

2.2、何时使用

定位（内核）函数位置
查看函数被调用时的调用堆栈、局部变量、参数
查看函数指针变量实际指的是哪个函数
查看代码的执行轨迹（哪些行被执行了）
查看内核或者进程的执行流程
调试内存泄露或者内存重复释放
统计函数调用次数
……

2.3、stap命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
stap [OPTIONS] FILENAME [ARGUMENTS]
stap [OPTIONS] - [ARGUMENTS]
stap [OPTIONS] –e SCRIPT [ARGUMENTS]

比较常用和有用的参数：
-e SCRIPT               Run given script.
-l PROBE                List matching probes.
-L PROBE                List matching probes and local variables.
-g                      guru mode 
-D NM=VAL               emit macro definition into generated C code
-o FILE                 send script output to file, instead of stdout.
-x PID                  sets target() to PID
-T time                 set execute time
-G var=val              set variable
-d object               add trace object

2.4、staprun命令

1
staprun [OPTIONS] MODULE [MODULE-OPTIONS]

stap命令与staprun命令的区别在于：
stap命令的操作对象是stp文件或script命令等，而staprun命令的操作对象是编译生成的内核模块。

3、脚本语言

3.1、probe

“probe” <=> “探测”, 是SystemTap进行具体地收集数据的关键字。“probe point” 是probe动作的时机，也称探测点。也就是probe程序监视的某事件点，一旦侦测的事件触发了，则probe将从此处插入内核或者用户进程中。“probe handle” 是当probe插入内核或者用户进程后所做的具体动作。
probe用法：

1
probe probe-point { statement }

在Hello World例子中begin和end就是probe-point， statement就是该探测点的处理逻辑，在Hello World例子中statement只有一行print，statement可以是复杂的代码块。
探测点语法：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
kernel.function(PATTERN)
kernel.function(PATTERN).call
kernel.function(PATTERN).return
kernel.function(PATTERN).return.maxactive(VALUE)
kernel.function(PATTERN).inline
kernel.function(PATTERN).label(LPATTERN)
module(MPATTERN).function(PATTERN)
module(MPATTERN).function(PATTERN).call
module(MPATTERN).function(PATTERN).return.maxactive(VALUE)
module(MPATTERN).function(PATTERN).inline
kernel.statement(PATTERN)
kernel.statement(ADDRESS).absolute
module(MPATTERN).statement(PATTERN)
process(PROCESSPATH).function(PATTERN)
process(PROCESSPATH).function(PATTERN).call
process(PROCESSPATH).function(PATTERN).return
process(PROCESSPATH).function(PATTERN).inline
process(PROCESSPATH).statement(PATTERN)

PATTERN语法为：

1
2
func[@file]
func@file:linenumber

例如：

1
2
3
4
kernel.function("*init*")
module("ext3").function("*")
kernel.statement("*@kernel/time.c:296")
process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request")

在return探测点可以用$return获取该函数的返回值。
inline函数无法安装.return探测点，也无法用$return获取其返回值。

3.2 基本语法

SystemTap脚本语法比较简单，与C语言类似，只是每一行结尾”;”是可选的。主要语句如下：
if/else、while、for/foreach、break/continue、return、next、delete、try/catch
其中：
next：主要在probe探测点逻辑处理中使用，调用此语句时，立刻从调用函数中退出。不同于exit()的是，next只是退出当前的调用函数，而此SystemTap并没有终了，但exit()则会终止SystemTap。

3.3 变量

不需要明确声明变量类型，脚本语言会根据函数参数等自动判断变量是什么类型的。
局部变量：在声明的probe和block（”{ }“范围内的部分）内有效。
全局变量：用”global“声明的变量，在此SystemTap的整个动作过程中都有效。全局变量的声明位置没有具体要求。需要注意的是，全局变量默认有锁保护，使用过多会有性能损失，如果用全局变量保存指针，可能出现指针所指的内容被进程修改，在探测点中拿不到真正的数据。
获取进程中的变量（全局变量、局部变量、参数）直接在变量名前面加$即可。

3.4 注释

1
2
3
# ...... ： Shell语言风格    
//...... ： C++语言风格    
 /*......*/ ： C语言风格

3.5 运算符

比较运算符、算数运算符基本上与C语言一样，需要特别指出的是：
(1)、.操作符：连接两个字符串，类似于php；
(2)、=和!：正则匹配和正则不匹配；

3.6 函数

函数的例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function indent:string (delta:long){
  return _generic_indent(-1, "",  delta)
}

function _generic_indent (idx, desc, delta)
{
  ts = __indent_timestamp ()
  if (! _indent_counters[idx]) _indent_timestamps[idx] = ts
  depth = _generic_indent_depth(idx, delta)
  return sprintf("%6d (%d:%d) %s:%-*s", (ts - _indent_timestamps[idx]), depth, delta, desc, depth, "")
}  

function strlen:long(s:string) %{
    STAP_RETURN(strlen(STAP_ARG_s));
%}

官方有很多很有用的函数，详情请参考：https://sourceware.org/systemtap/tapsets/
以及在本机安装了SystemTap之后在目录/usr/local/share/systemtap/tapset/下也可以看具体函数的实现以及一些奇特的用法。

3.7、技巧

3.7.1、定位函数位置

在一个大型项目中找出函数在哪里定义有时很有用，特别是一些比较难找出在哪里定义的函数，比如内核或者glibc中的某个函数想要看其实现时，首先得找出其在哪个文件的哪一行定义，用SystemTap一行命令就可以搞定。
比如要看printf在glibc中哪里定义的：

1
2
root@j9 ~# stap -l 'kernel.function("sys_recv")'
kernel.function("sys_recv@/build/buildd/linux-lts-trusty-3.13.0/net/socket.c:1868")

可以看出recv是在socket.c第1868行定义的。
甚至可以*号来模糊查找：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
root@j9 ~# stap -l 'kernel.function("*recv")'   
kernel.function("__audit_mq_sendrecv@/build/buildd/linux-lts-trusty-3.13.0/kernel/auditsc.c:2062")
kernel.function("audit_mq_sendrecv@/build/buildd/linux-lts-trusty-3.13.0/include/linux/audit.h:263")
kernel.function("compat_sys_recv@/build/buildd/linux-lts-trusty-3.13.0/net/compat.c:762")
kernel.function("i2c_master_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/i2c/i2c-core.c:1827")
kernel.function("ip_cmsg_recv@/build/buildd/linux-lts-trusty-3.13.0/net/ipv4/ip_sockglue.c:147")
kernel.function("kgdb_tty_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/tty/serial/kgdb_nmi.c:109")
kernel.function("ppp_do_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/net/ppp/ppp_generic.c:1617")
kernel.function("scm_recv@/build/buildd/linux-lts-trusty-3.13.0/include/net/scm.h:109")
kernel.function("sys_recv@/build/buildd/linux-lts-trusty-3.13.0/net/socket.c:1868")
kernel.function("tcp_event_data_recv@/build/buildd/linux-lts-trusty-3.13.0/net/ipv4/tcp_input.c:615")
kernel.function("tcp_splice_data_recv@/build/buildd/linux-lts-trusty-3.13.0/net/ipv4/tcp.c:637")
kernel.function("tpm_tis_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/char/tpm/tpm_tis.c:231")
kernel.function("try_fill_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/net/virtio_net.c:615")

同理，也可以用来定位用户进程的函数位置：
比如tengine的文件ngx_shmem.c里面为了兼容各个操作系统而实现了三个版本的ngx_shm_alloc，用#if (NGX_HAVE_MAP_ANON)、#elif (NGX_HAVE_MAP_DEVZERO)、#elif (NGX_HAVE_SYSVSHM)、#endif来做条件编译，那怎么知道编译出来的是哪个版本呢，用SystemTap的话就很简单了，否则要去grep一下这几宏有没有定义才知道了。

1
2
[root@cache4 tengine]# stap -l 'process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc")'
process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc@src/os/unix/ngx_shmem.c:15")

3.7.2 查看可用探测点以及该探测点上可用的变量

在一些探测点上能获取的变量比较有限，这是因为这些变量可能已经被编译器优化掉了，优化掉的变量就获取不到了。一般先用-L参数来看看有哪些变量可以直接使用：

1
2
[root@cache4 tengine]# stap -L 'process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc")' 
process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc@src/os/unix/ngx_shmem.c:15") $shm:ngx_shm_t*

可见在该探测点上可以直接使用$shm这个变量，其类型是ngx_shm_t*。
statement探测点也类似：

1
2
3
4
5
[root@cache4 tengine]# stap -L 'process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:*")'                   
process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:395") $pool:ngx_pool_t* $size:size_t
process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:398") $pool:ngx_pool_t* $size:size_t
process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:399") $size:size_t
process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:404") $size:size_t $p:void*

可以直接使用这些探测点

3.7.3 输出调用堆栈

用户态探测点堆栈：print_ubacktrace()、sprint_ubacktrace()
内核态探测点堆栈：print_backtrace()、sprint_backtrace()
不带s和带s的区别是前者直接输出，后者是返回堆栈字符串。
这几个函数非常有用，在排查问题时可以根据一些特定条件来过滤函数被执行时是怎么调用进来的，比如排查tengine返回5xx时的调用堆栈是怎样的：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#cat debug_tengine_5xx.stp 
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_finalize_request").call {
    if ($rc >= 500) {
        printf("rc: %d\n", $rc)
        print_ubacktrace()
    }
}

#stap debug_tengine_5xx.stp 
rc: 502
 0x49af2e : ngx_http_finalize_request+0xe/0x480 [/home/admin/tengine/bin/nginx]
 0x543305 : ngx_http_video_flv_send_rest+0xf5/0x380 [/home/admin/tengine/bin/nginx]
 0x543187 : ngx_http_video_finalize_request+0x57/0xe0 [/home/admin/tengine/bin/nginx]
 0x49828f : ngx_http_terminate_request+0x4f/0xc0 [/home/admin/tengine/bin/nginx]
 0x49b760 : ngx_http_test_reading+0x50/0x130 [/home/admin/tengine/bin/nginx]
 0x49779f : ngx_http_request_handler+0x1f/0x40 [/home/admin/tengine/bin/nginx]
 0x47ea8f : ngx_epoll_process_events+0x2df/0x330 [/home/admin/tengine/bin/nginx]
 0x4753f9 : ngx_process_events_and_timers+0x69/0x1c0 [/home/admin/tengine/bin/nginx]
 0x47d4d8 : ngx_worker_process_cycle+0x138/0x260 [/home/admin/tengine/bin/nginx]
 0x47a38a : ngx_spawn_process+0x1ca/0x5e0 [/home/admin/tengine/bin/nginx]
 0x47c73c : ngx_start_worker_processes+0x7c/0x100 [/home/admin/tengine/bin/nginx]
 0x47db5f : ngx_master_process_cycle+0x3af/0x9b0 [/home/admin/tengine/bin/nginx]
 0x45a740 : main+0xa90/0xb50 [/home/admin/tengine/bin/nginx]
 0x3623e1ecdd [/lib64/libc-2.12.so+0x1ecdd/0x38d000]

比如看看内核是怎么收包的：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
root@jusse ~# cat netif_receive_skb.stp 
probe kernel.function("netif_receive_skb") 
{ 
    printf("--------------------------------------------------------\n"); 
    print_backtrace(); 
    printf("--------------------------------------------------------\n"); 
} 
root@jusse ~# stap netif_receive_skb.stp
--------------------------------------------------------
 0xffffffff8164dc00 : netif_receive_skb+0x0/0x90 [kernel]
 0xffffffff8164e280 : napi_gro_receive+0xb0/0x130 [kernel]
 0xffffffff81554537 : handle_incoming_queue+0xe7/0x100 [kernel]
 0xffffffff815555d9 : xennet_poll+0x279/0x430 [kernel]
 0xffffffff8164ee09 : net_rx_action+0x139/0x250 [kernel]
 0xffffffff810702cd : __do_softirq+0xdd/0x300 [kernel]
 0xffffffff8107088e : irq_exit+0x11e/0x140 [kernel]
 0xffffffff8144e785 : xen_evtchn_do_upcall+0x35/0x50 [kernel]
 0xffffffff8176c9ed : xen_hvm_callback_vector+0x6d/0x80 [kernel]
--------------------------------------------------------

3.7.4 获取函数参数

一些被编译器优化掉的函数参数用-L去看的时候没有找到，这样的话在探测点里面也不能直接用$方式获取该参数变量，这时可以使用SystemTap提供的_arg函数接口，是根据类型指定的，比如pointer_arg是获取指针类型参数，int_arg是获取整型参数，类似的还有long_arg、longlong_arg、uint_arg、ulong_arg、ulonglong_arg、s32_arg、s64_arg、u32_arg、u64_arg：

1
2
3
4
5
6
7
8
9
10
11
12
13
root@j9 ~# stap -L 'kernel.function("sys_open")' 
kernel.function("SyS_open@/build/buildd/linux-lts-trusty-3.13.0/fs/open.c:1011") $ret:long int
root@j9 ~# cat sys_open.stp 
probe kernel.function("sys_open").call
{
    printf("filename: %p(%s), flags: %d, mode: %x\n", pointer_arg(1), kernel_string(pointer_arg(1)), int_arg(2), int_arg(3));
}
root@j9 ~# stap sys_open.stp 
filename: 0xc2081d2120(/proc/stat), flags: 524288, mode: 0
filename: 0x7facec00e838(/root/opt/libexec/systemtap/stapio), flags: 0, mode: 1b6
filename: 0x2219488(/var/log/auth.log), flags: 0, mode: 1b6
filename: 0x7facec00e838(/root/opt/libexec/systemtap/stapio), flags: 0, mode: 1b6
filename: 0x7fad10172c29(/etc/passwd), flags: 524288, mode: 1b6

这两个函数的参数完全兼容，只是第二个参数命名不一样而已，可以像下面这么用：

1
2
3
4
5
6
7
8
#cat debug_tengine_5xx.stp 
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_finalize_request").call, process("/home/admin/tengine/bin/nginx").function("ngx_http_special_response_handler").call {
    rc = int_arg(2)
    if (rc >= 500) {
        printf("rc: %d\n", rc)
        print_ubacktrace()
    }
}

3.7.5 获取全局变量

有时候用$可以直接获取到全局变量，但有时候又获取不到，那可以试试@var：
比如获取nginx的全局变量ngx_cycyle：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@j9 ~# cat get_ngx_cycle.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers").call {
    printf("ngx_cycle->connections: %d\n", $ngx_cycle->connections)
    exit()
}

root@j9 ~# stap get_ngx_cycle.stp
semantic error: while processing probe process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers@src/event/ngx_event.c:225").call from: process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers").call

semantic error: unable to find local 'ngx_cycle', [man error::dwarf] dieoffset 0x73ca8 in /home/admin/tengine/bin/nginx, near pc 0x434152 in ngx_process_events_and_timers src/event/ngx_event.c (alternatives: $cycle, $delta, $timer, $flags)): identifier '$ngx_cycle' at get_ngx_cycle.stp:3:44
        source:     printf("ngx_cycle->connections: %d\n", $ngx_cycle->connections)
                                                           ^

Pass 2: analysis failed.  [man error::pass2]

root@j9 ~# cat get_ngx_cycle.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers").call {
    ngx_cycle = @var("ngx_cycle@src/core/ngx_cycle.c")
    printf("ngx_cycle->connections: %d\n", ngx_cycle->connections)
    exit()
}

root@j9 ~# stap get_ngx_cycle.stp
ngx_cycle->connections: 19507312

3.7.6 获取数据结构成员用法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#cat get_http_uri.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call {
    printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data)
}
#stap get_http_uri.stp
WARNING: never-assigned local variable 'len' (similar: data): identifier 'len' at get_http_uri.stp:2:57
 source:     printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data)
                                                                 ^
WARNING: never-assigned local variable 'data' (similar: len): identifier 'data' at :2:70
 source:     printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data)
                                                                              ^
semantic error: invalid operator: operator '.' at :2:56
        source:     printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data)
                                                                       ^

semantic error: type mismatch: expected long but found string: operator '.' at :2:56
        source:     printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data)
                                                                       ^

Pass 2: analysis failed.  [man error::pass2]

#cat get_http_uri.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call {
    printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri->len, $r->uri->data)
}

#stap get_http_uri.stp
r->uri.len: 1, r->uri.data: 0x1276f94
r->uri.len: 1, r->uri.data: 0x11d5fc4
r->uri.len: 1, r->uri.data: 0x124fd24
^C

3.7.7 输出整个数据结构

SystemTap有两个语法可以输出整个数据结构：在变量的后面加一个或者两个$即可，例子如下：

1
2
3
4
5
6
7
#cat get_r_pool.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call {
    printf("$r->pool$: %s\n$r->pool$$: %s\n", $r->pool$, $r->pool$$)
}
#stap get_r_pool.stp
$r->pool$: {.d={...}, .max=4016, .current=0x161acd0, .chain=0x0, .large=0x0, .cleanup=0x0, .log=0x161c690}
$r->pool$$: {.d={.last="a", .end="", .next=0x1617650, .failed=0}, .max=4016, .current=0x161acd0, .chain=0x0, .large=0x0, .cleanup=0x0, .log=0x161c690}

其中r->pool的结构如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
typedef struct {
    u_char               *last;
    u_char               *end;
    ngx_pool_t           *next;
    ngx_uint_t            failed;
} ngx_pool_data_t;

struct ngx_pool_s {
    ngx_pool_data_t       d;
    size_t                max;
    ngx_pool_t           *current;
    ngx_chain_t          *chain;
    ngx_pool_large_t     *large;
    ngx_pool_cleanup_t   *cleanup;
    ngx_log_t            *log;
#if  (NGX_DEBUG_POOL)
    size_t                size;
    ngx_pool_stat_t      *stat;
#endif
};

ngx_pool_s包含了结构ngx_pool_data_t。变量后面加和$的区别是后者展开了里面的结构而前者不展开，此用法只输出基本数据类型的值。

3.7.8 输出字符串指针

用户态使用：user_string、user_string_n
内核态使用：kernel_string、kernel_string_n、user_string_quoted

1
2
3
4
5
6
7
8
9
#cat get_http_uri.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call {
    printf("r->uri: %s\nr->uri(n): %s\n", user_string($r->uri->data), user_string_n($r->uri->data, $r->uri->len))
}

#stap get_http_uri.stp
r->uri: /?id=1 HTTP/1.1
User-Agent
r->uri(n): /

user_string_quoted是获取用户态传给内核的字符串，代码中一般有__user宏标记：

1
2
3
4
5
6
7
8
9
#cat sys_open.stp
probe kernel.function("sys_open")
{
    printf("filename: %s\n", user_string_quoted(pointer_arg(1)));
}
#stap sys_open.stp 
filename: "/var/log/auth.log"
filename: "/proc/stat"
filename: "/proc/uptime"

3.7.9 指针类型转换

SystemTap提供@cast来实现指针类型转换，比如可以将void *转成自己需要的类型

1
2
3
4
5
6
7
8
9
10
11
12
#cat get_c_fd.stp 
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request_line").call {
    printf("c->fd: %d\n", @cast($rev->data, "ngx_connection_t")->fd)
}

#stap get_c_fd.stp 
c->fd: 3
c->fd: 28
c->fd: 30
c->fd: 32
c->fd: 34
^C

3.7.10 定义某个类型的变量

同样是用@cast，定义一个变量用来保存其转换后的地址即可，用法如下：

1
2
3
4
5
6
7
8
9
10
11
#cat get_c.stp 
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request_line").call {
    c = &@cast($rev->data, "ngx_connection_t")
    printf("c->fd: %d, c->requests: %d\n", c->fd, c->requests)
}

#stap get_c.stp 
c->fd: 3, c->requests: 1
c->fd: 28, c->requests: 1
c->fd: 30, c->requests: 1
^C

3.7.11 多级指针用法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
root@j9 ~# cat cc_multi_pointer.c
#include <stdio.h>

struct test {
    int count;
};

int main(int argc, char *argv[])
{   
    struct test t = {.count = 5566};
    struct test *pt = &t;
    struct test **ppt = &pt;

    printf("t.count: %d, pt->count: %d, ppt->count: %d\n", t.count, pt->count, (*ppt)->count);

    return 0;
}

root@j9 ~# gcc -Wall -g -o cc_multi_pointer ./cc_multi_pointer.c

root@j9 ~# cat cc_multi_pointer.stp
probe process("./cc_multi_pointer").statement("main@./cc_multi_pointer.c:13")
{   
    printf("$t->count: %d, $pt->count: %d, $ppt->count: %d", $t->count, $pt->count, $ppt[0]->count);
}

root@j9 ~# ./cc_multi_pointer
t.count: 5566, pt->count: 5566, ppt->count: 5566

root@j9 ~# stap ./cc_multi_pointer.stp -c './cc_multi_pointer'
t.count: 5566, pt->count: 5566, ppt->count: 5566
$t->count: 5566, $pt->count: 5566, $ppt->count: 5566

简言之：通过[0]去解引用即可

3.7.12 遍历C语言数组

下面是在nginx处理请求关闭时遍历请求头的例子：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#cat debug_http_header.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_finalize_request").call {
    i = 0
    headers_in_part = &$r->headers_in->headers->part
    headers = &@cast(headers_in_part->elts, "ngx_table_elt_t")[0]
    while (headers) {
        if (i >= headers_in_part->nelts) {
            if (!headers_in_part->next) {
                break
            }
            headers_in_part = headers_in_part->next;
            headers = &@cast(headers_in_part->elts, "ngx_table_elt_t")[0]
            i = 0
        }
        h = &@cast(headers, "ngx_table_elt_t")[i]
        printf("%s: %s\n", user_string_n(h->key->data, h->key->len), user_string_n(h->value->data, h->value->len))
        i += 1
    }
}

#stap debug_http_header.stp
User-Agent: curl/7.29.0
Host: 127.0.0.1:20090
Accept: */*

3.7.13 查看函数指针所指的函数名

获取一个地址所对应的符号：
用户态：usymname
内核态：symname

1
2
3
4
5
6
7
8
9
#cat get_c_handler.stp
probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request_line").call {
    c = &@cast($rev->data, "ngx_connection_t")
    printf("c->read->handlers: %s, c->write->handler: %s\n", usymname(c->read->handler), usymname(c->write->handler))
}

#stap get_c_handler.stp
c->read->handlers: ngx_http_process_request_line, c->write->handler: ngx_http_empty_handler
^C

3.7.14 修改进程中的变量

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
root@j9 ~# cat stap_set_var.c -n     
     1  #include <stdio.h>
     2
     3  typedef struct policy {
     4      int     id;
     5  } policy_t;
     6
     7  int main(int argc, char *argv[])
     8  {
     9      policy_t policy;
    10      policy_t *p = &policy;
    11      policy_t **pp;
    12
    13      p->id = 111;
    14
    15      printf("before stap set, p->id: %d\n", p->id);
    16
    17      pp = &p;
    18
    19      printf("after stap set, p->id: %d, (*pp)->id: %d\n", p->id, (*pp)->id);
    20
    21      return 0;
    22  }

root@j9 ~# gcc -Wall -g -o ./stap_set_var ./stap_set_var.c

root@j9 ~# cat stap_set_var.stp
probe process("./stap_set_var").statement("main@./stap_set_var.c:17")
{
    $p->id = 222;
    printf("$p$: %s\n", $p$)
}

root@j9 ~# stap -g stap_set_var.stp -c ./stap_set_var
before stap set, p->id: 111
after stap set, p->id: 222, (*pp)->id: 222
$p$: {.id=222}

可以看出在第17行用SystemTap修改后的值在第19行就生效了。
需要注意的是stap要加-g参数在guru模式下才能修改变量的值。

3.7.15 跟踪进程执行流程

thread_indent(n): 补充空格
ppfunc(): 当前探测点所在的函数
在call探测点调用thread_indent(4)补充4个空格，在return探测点调用thread_indent(-4)回退4个空格，效果如下：

1
2
3
4
5
6
7
8
9
10
#cat trace_nginx.stp
probe process("/home/admin/tengine/bin/nginx").function("*@src/http/ngx_http_*").call
{
    printf("%s -> %s\n", thread_indent(4), ppfunc());
}

probe process("/home/admin/tengine/bin/nginx").function("*@src/http/ngx_http_*").return
{
    printf("%s <- %s\n", thread_indent(-4), ppfunc());
}

3.7.16 查看代码执行路径

pp(): 输出当前被激活的探测点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
 
#cat ngx_http_process_request.stp
probe process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:*") {
    printf("%s\n", pp())
}

#stap ngx_http_process_request.stp 
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2762")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2768")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2771")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2773")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2774")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2783")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2835")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2840")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2841")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2842")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2843")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2846")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2847")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2848")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2850")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2852")
process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2853")
^C

可以看出该函数哪些行被执行了

3.7.17 巧用正则匹配过滤

在排查问题时，可以利用一些正则匹配来获取自己想要的信息，比如下面是只收集*.j9.com的堆栈：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
#
cat debug_tengine_5xx.stp 
probe process("/home/admin/tengine/bin/t-coresystem-tengine-cdn").function("ngx_http_finalize_request").call {
    rc = $rc
    if (rc < 0) {
        host = "(null)"
        if ($r->headers_in->server->len != 0) {
            host = user_string_n($r->headers_in->server->data, $r->headers_in->server->len)
        } else {
            cscf = &@cast($r->srv_conf, "ngx_http_core_srv_conf_t")[@var("ngx_http_core_module@src/http/ngx_http_core_module.c")->ctx_index]
            if (cscf->server_name->len != 0) {
                 host = user_string_n(cscf->server_name->data, cscf->server_name->len)
            }
        }

        if (host =~ ".*\.j9\.com") {
            printf("rc: %d, host: %s\n", rc, host)
            print_ubacktrace()
        }
    }
    
ss#stap debug_tengine_5xx.stp
WARNING: Missing unwind data for module, rerun with 'stap -d /lib64/libc-2.12.so'
rc: -4, host: www.j9.com
 0x49af2e : ngx_http_finalize_request+0xe/0x480 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x492eab : ngx_http_core_content_phase+0x2b/0x130 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x48e74d : ngx_http_core_run_phases+0x3d/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x514c3c : ngx_http_lua_socket_tcp_read+0x44c/0x590 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x513150 : ngx_http_lua_socket_tcp_handler+0x30/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x475b96 : ngx_event_process_posted+0x36/0x40 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47d4d8 : ngx_worker_process_cycle+0x138/0x260 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47a38a : ngx_spawn_process+0x1ca/0x5e0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47c73c : ngx_start_worker_processes+0x7c/0x100 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47db5f : ngx_master_process_cycle+0x3af/0x9b0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x45a740 : main+0xa90/0xb50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x3623e1ecdd [/lib64/libc-2.12.so+0x1ecdd/0x38d000]
rc: -4, host: cdn.j9.com
 0x49af2e : ngx_http_finalize_request+0xe/0x480 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x492eab : ngx_http_core_content_phase+0x2b/0x130 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x48e74d : ngx_http_core_run_phases+0x3d/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x514c3c : ngx_http_lua_socket_tcp_read+0x44c/0x590 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x513150 : ngx_http_lua_socket_tcp_handler+0x30/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x475b96 : ngx_event_process_posted+0x36/0x40 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47d4d8 : ngx_worker_process_cycle+0x138/0x260 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47a38a : ngx_spawn_process+0x1ca/0x5e0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47c73c : ngx_start_worker_processes+0x7c/0x100 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x47db5f : ngx_master_process_cycle+0x3af/0x9b0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x45a740 : main+0xa90/0xb50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn]
 0x3623e1ecdd [/lib64/libc-2.12.so+0x1ecdd/0x38d000]
}




3.7.18 关联数组用法

SystemTap的关联数组必须是全局变量，需要用global进行声明，其索引可以支持多达9项索引域,各域间以逗号隔开。支持 =, ++ 与 +=操作,其默认的初始值为0。
例如：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
root@j9 ~# cat stap_array.stp 
global reads
probe vfs.read {
    reads[execname(), pid()] ++
}
probe timer.s(3) {
    foreach ([execname, pid] in reads) {
        printf("%s(%d) : %d \n", execname, pid, reads[execname, pid])
    }
    print("============================\n")
    delete reads
}

root@j9 ~# stap stap_array.stp 
stapio(18716) : 16 
rsyslogd(770) : 1 
docker(743) : 3 
IFSWatch(5594) : 30 
QThread(5594) : 6 
AliYunDunUpdate(1057) : 4 
sshd(15118) : 1 
sshd(15191) : 1 
============================
stapio(18716) : 16 
sshd(15191) : 3 
docker(743) : 3 
IFSWatch(5594) : 30 
sshd(15118) : 2 
QThread(5594) : 12 
AliYunDunUpdate(1057) : 8 
============================
^C
root@j9 ~/systemtap#

也可以用+、-进行排序：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
root@j9 ~# cat stap_array.stp
global reads
probe vfs.read {
    reads[execname(), pid()] ++
}
probe timer.s(3) {
    foreach ([execname, pid+] in reads) {
        printf("%s(%d) : %d \n", execname, pid, reads[execname, pid])
    }
    print("============================\n")
    delete reads
}

root@j9 ~# stap stap_array.stp 
docker(743) : 3 
rsyslogd(770) : 1 
AliYunDunUpdate(1057) : 12 
IFSWatch(5594) : 30 
QThread(5594) : 12 
sshd(15118) : 2 
sshd(15191) : 2 
stapio(19021) : 16 
============================
docker(743) : 3 
AliYunDunUpdate(1057) : 12 
IFSWatch(5594) : 30 
QThread(5594) : 6 
sshd(15118) : 1 
sshd(15191) : 19 
stapio(19021) : 16 
============================
^C
root@j9 ~#

3.7.19 调试内存泄漏以及内存重复释放

在return探测点，使用函数入参需要@entry

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
probe begin {
    printf("=============begin============\n")
}

//记录内存分配和释放的计数关联数组
global g_mem_ref_tbl
//记录内存分配和释放的调用堆栈关联数组，以内存地址作为key
global g_mem_bt_tbl

probe process("/lib/x86_64-linux-gnu/libc.so.6").function("__libc_malloc").return, process("/lib/x86_64-linux-gnu/libc.so.6").function("__libc_calloc").return {
    if (target() == pid()) {
        if (g_mem_ref_tbl[$return] == 0) {
            g_mem_ref_tbl[$return]++
            g_mem_bt_tbl[$return] = sprint_ubacktrace()
        }
    }
}
//$mem是__libc_free的参数
probe process("/lib/x86_64-linux-gnu/libc.so.6").function("__libc_free").call {
    if (target() == pid()) {
        g_mem_ref_tbl[$mem]--

        if (g_mem_ref_tbl[$mem] == 0) {
            if ($mem != 0) {
                //记录上次释放的调用堆栈
                g_mem_bt_tbl[$mem] = sprint_ubacktrace()
            }
        } else if (g_mem_ref_tbl[$mem] < 0 && $mem != 0) {
            //如果调用free已经失衡，那就出现了重复释放内存的问题，这里输出当前调用堆栈，以及这个地址上次释放的调用堆栈
            printf("MMMMMMMMMMMMMMMMMMMMMMMMMMMM\n")
            printf("g_mem_ref_tbl[%p]: %d\n", $mem, g_mem_ref_tbl[$mem])
            print_ubacktrace()
            printf("last free backtrace:\n%s\n", g_mem_bt_tbl[$mem])
            printf("WWWWWWWWWWWWWWWWWWWWWWWWWWWW\n")
        }
    }
}

probe end {
    //最后输出产生泄漏的内存是在哪里分配的
    printf("=============end============\n")
    foreach(mem in g_mem_ref_tbl) {
        if (g_mem_ref_tbl[mem] > 0) {
            printf("%s\n", g_mem_bt_tbl[mem])
            printf("--------------------------------\n")
        }
    }
}

3.7.20 嵌入C代码

在进程fork出子进程时打印出进程id和进程名:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
root@jusse ~/systemtap# cat copy_process.stp
function getprocname:string(task:long)
%{
    struct task_struct *task = (struct task_struct *)STAP_ARG_task;
    snprintf(STAP_RETVALUE, MAXSTRINGLEN, "pid: %d, comm: %s", task->pid, task->comm);
%}

function getprocid:long(task:long)
%{
    struct task_struct *task = (struct task_struct *)STAP_ARG_task;
    STAP_RETURN(task->pid);
%}

probe kernel.function("copy_process").return
{
    printf("copy_process return: %p, pid: %d, getprocname: %s, getprocid: %d\n", $return, $return->pid, getprocname($return), getprocid($return));
}

root@jusse ~/systemtap# stap -g copy_process.stp
copy_process return: 0xffff880039f61800, pid: 12212, getprocname: pid: 12212, comm: bash, getprocid: 12212
copy_process return: 0xffff880039f61800, pid: 12212, getprocname: pid: 12212, comm: bash, getprocid: 12212
copy_process return: 0xffff880039f63000, pid: 12213, getprocname: pid: 12213, comm: cc_epoll, getprocid: 12213
copy_process return: 0xffff880039f63000, pid: 12213, getprocname: pid: 12213, comm: cc_epoll, getprocid: 12213
copy_process return: 0xffff8800081a9800, pid: 12214, getprocname: pid: 12214, comm: cc_epoll, getprocid: 12214
copy_process return: 0xffff8800081a9800, pid: 12214, getprocname: pid: 12214, comm: cc_epoll, getprocid: 12214
copy_process return: 0xffff8800004d8000, pid: 12215, getprocname: pid: 12215, comm: cc_epoll, getprocid: 12215
copy_process return: 0xffff8800004d8000, pid: 12215, getprocname: pid: 12215, comm: cc_epoll, getprocid: 12215
copy_process return: 0xffff880000564800, pid: 12216, getprocname: pid: 12216, comm: cc_epoll, getprocid: 12216
copy_process return: 0xffff880000564800, pid: 12216, getprocname: pid: 12216, comm: cc_epoll, getprocid: 12216
copy_process return: 0xffff880000566000, pid: 12217, getprocname: pid: 12217, comm: cc_epoll, getprocid: 12217
copy_process return: 0xffff880000566000, pid: 12217, getprocname: pid: 12217, comm: cc_epoll, getprocid: 12217

有三个需要注意的地方：
1）、SystemTap脚本里面嵌入C语言代码要在每个大括号前加%前缀，是%{…… %} 而不是%{ …… }%；
2）、获取脚本函数参数要用STAP_ARG_前缀；
3）、一般long等返回值用STAP_RETURN，而string类型返回值要用snprintf、strncat等方式把字符串复制到STAP_RETVALUE里面。

3.7.21 调试内核模块

这小节就不细讲了，这篇博客 (http://blog.chinaunix.net/uid-14528823-id-4726046.html) 写得很详细，这里只copy两个关键点过来记录一下：
要调试自己的内核模块，需要注意的有两个关键点：
1)、使用SystemTap调试内核模块，探测点的编写格式示例为：
module(“ext3”).function(“ext3_*”)
2)、需要将自己的模块cp到/lib/modules/uname -r/extra目录中，否则找不到符号，如果/lib/modules/uname -r/目录下没有extra这个目录，自己mkdir一下就可以。

3.7.22 一些错误提示及解决办法

错误提示1：

1
2
ERROR: MAXACTION exceeded near keyword at debug_connection.stp:86:9
ERROR: MAXACTION exceeded near operator '->' at debug_connection.stp:84:30

解决办法：
加上stap参数：-DMAXACTION=102400，如果还报这种类型的错误，只需把102400调成更大的值即可。
错误提示2：

1
WARNING: Number of errors: 0, skipped probes: 82

解决办法：
加上-DMAXSKIPPED=102400和-DSTP_NO_OVERLOAD参数
还有一些可以去掉限制的宏：

MAXSTRINGLEN：这个宏会影响sprintf的buffer大小，默认为512字节。
MAXTRYLOCK：对全局变量进行try lock操作的次数，超过则次数还拿不到锁则放弃和跳过该探测点，默认值为1000.全局变量多的时候可以把这个宏开大一点。

3.7.23 传递参数

3.7.24 常用函数

本节来介绍systemtap中常用的一些函数
tid()：当前线程ID。
uid()：当前用户ID。
cpu()：当前CPU编号。
ctime()：当前UNIX epoch秒数。
pp()：当前探测点的描述字符串
exit(): 执行一次后退出。
execname()：当前运行的进程名称。
probefunc()：探测点函数名称。
target()：在stap使用-c command或者-x process命令时，target()能拿到进程的pid。
name()：返回系统调用的名称字符串，仅能在syscall类型的探针处理函数中使用。
thread_indent(delta)：它可以输出当前probe所处的可执行程序名称、线程id、函数执行的相对时间和执行的次数（通过空格的数量）信息，它的返回值就是一个字符串。参数delta是在每次调用时增加或移除的空白数量。
@defined和@choose_defined
由于版本变化，有一些变量可能在新版本中不存在了，此时可以使用@define来检查变量是否存在：

1
2
3
4
5
probe vm.pagefault = kernel.function("__handle_mm_fault@mm/memory.c") ?,
                     kernel.function("handle_mm_fault@mm/memory.c") ?
{
  write_access = (@defined($flags) ? $flags & FAULT_FLAG_WRITE : $write_access)
}

@entry
在.return探针中，有一个特殊的操作符@entry，用于存储该探针的入口处的表达式的值，可以使用这个操作符，完成比如计算探针函数执行时间计算等工作，比如：

1
2
3
4
5
6
7
global sloth = 50
      
probe vfs.open.return {
  time = gettimeofday_us()-@entry(gettimeofday_us())
  if (time >= sloth)
    printf("%s[%d] %d %s\n", execname(), tid(), time, pathname)
}

这个脚本在vfs.open.return探针处理函数中，通过@entry操作符，计算完成vfs.open操作的时间差，如果超过设置的阈值50就打印相关信息。
-G命令行参数，可以设置全局变量VAR的值为VAL，相应地就可以作为开关来控制脚本的行为，比如：

1
2
3
4
5
6
7
8
9
10
11
// sudo stap G-params.stp -G flag=1
// flag has set
global flag=0

probe begin {
  if (flag == 0) {
    printf("flag not set\n")
  } else {
    printf("flag has set\n")
  }
}

3.7.25 异步事件

常见的异步事件是begin、end、never、timers。
timers用于定义定时器探测点，常见的格式timer.s(1)来定义每秒触发的探测点。
never定义的探测点不会被调用到，很多时候加这个探测点只是为了检查一些语法错误。

Nginx

Nginx的http模块代码分析

1、代码框架

http模块主要代码放在ngx_http.c这个文件中，接下来我们就来分析下相关代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
static char *ngx_http_block(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
static ngx_int_t ngx_http_init_phases(ngx_conf_t *cf,
    ngx_http_core_main_conf_t *cmcf);
static ngx_int_t ngx_http_init_headers_in_hash(ngx_conf_t *cf,
    ngx_http_core_main_conf_t *cmcf);
static ngx_int_t ngx_http_init_phase_handlers(ngx_conf_t *cf,
    ngx_http_core_main_conf_t *cmcf);

static ngx_int_t ngx_http_add_addresses(ngx_conf_t *cf,
    ngx_http_core_srv_conf_t *cscf, ngx_http_conf_port_t *port,
    ngx_http_listen_opt_t *lsopt);
static ngx_int_t ngx_http_add_address(ngx_conf_t *cf,
    ngx_http_core_srv_conf_t *cscf, ngx_http_conf_port_t *port,
    ngx_http_listen_opt_t *lsopt);
static ngx_int_t ngx_http_add_server(ngx_conf_t *cf,
    ngx_http_core_srv_conf_t *cscf, ngx_http_conf_addr_t *addr);

static char *ngx_http_merge_servers(ngx_conf_t *cf,
    ngx_http_core_main_conf_t *cmcf, ngx_http_module_t *module,
    ngx_uint_t ctx_index);
static char *ngx_http_merge_locations(ngx_conf_t *cf,
    ngx_queue_t *locations, void **loc_conf, ngx_http_module_t *module,
    ngx_uint_t ctx_index);
static ngx_int_t ngx_http_init_locations(ngx_conf_t *cf,
    ngx_http_core_srv_conf_t *cscf, ngx_http_core_loc_conf_t *pclcf);
static ngx_int_t ngx_http_init_static_location_trees(ngx_conf_t *cf,
    ngx_http_core_loc_conf_t *pclcf);
static ngx_int_t ngx_http_cmp_locations(const ngx_queue_t *one,
    const ngx_queue_t *two);
static ngx_int_t ngx_http_join_exact_locations(ngx_conf_t *cf,
    ngx_queue_t *locations);
static void ngx_http_create_locations_list(ngx_queue_t *locations,
    ngx_queue_t *q);
static ngx_http_location_tree_node_t *
    ngx_http_create_locations_tree(ngx_conf_t *cf, ngx_queue_t *locations,
    size_t prefix);

static ngx_int_t ngx_http_optimize_servers(ngx_conf_t *cf,
    ngx_http_core_main_conf_t *cmcf, ngx_array_t *ports);
static ngx_int_t ngx_http_server_names(ngx_conf_t *cf,
    ngx_http_core_main_conf_t *cmcf, ngx_http_conf_addr_t *addr);
static ngx_int_t ngx_http_cmp_conf_addrs(const void *one, const void *two);
static int ngx_libc_cdecl ngx_http_cmp_dns_wildcards(const void *one,
    const void *two);

static ngx_int_t ngx_http_init_listening(ngx_conf_t *cf,
    ngx_http_conf_port_t *port);
static ngx_listening_t *ngx_http_add_listening(ngx_conf_t *cf,
    ngx_http_conf_addr_t *addr);
static ngx_int_t ngx_http_add_addrs(ngx_conf_t *cf, ngx_http_port_t *hport,
    ngx_http_conf_addr_t *addr);
#if (NGX_HAVE_INET6)
static ngx_int_t ngx_http_add_addrs6(ngx_conf_t *cf, ngx_http_port_t *hport,
    ngx_http_conf_addr_t *addr);
#endif

在ngx_http.c文件最前面这部分代码全部是函数声明，接下来看下http模块配置，指令设置、函数指针以及模块结构初始化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
ngx_uint_t   ngx_http_max_module;


ngx_http_output_header_filter_pt  ngx_http_top_header_filter;
ngx_http_output_body_filter_pt    ngx_http_top_body_filter;
ngx_http_request_body_filter_pt   ngx_http_top_request_body_filter;

ngx_str_t  ngx_http_html_default_types[] = {
    ngx_string("text/html"),
    ngx_null_string
};

// http指令定义
static ngx_command_t  ngx_http_commands[] = {

    { ngx_string("http"),
      NGX_MAIN_CONF|NGX_CONF_BLOCK|NGX_CONF_NOARGS,
      ngx_http_block,
      0,
      0,
      NULL },

      ngx_null_command
};

//核心模块上下文
static ngx_core_module_t  ngx_http_module_ctx = {
    ngx_string("http"),
    NULL,
    NULL
};

//http模块定义
ngx_module_t  ngx_http_module = {
    NGX_MODULE_V1,
    &ngx_http_module_ctx,                  /* module context */
    ngx_http_commands,                     /* module directives */
    NGX_CORE_MODULE,                       /* module type */
    NULL,                                  /* init master */
    NULL,                                  /* init module */
    NULL,                                  /* init process */
    NULL,                                  /* init thread */
    NULL,                                  /* exit thread */
    NULL,                                  /* exit process */
    NULL,                                  /* exit master */
    NGX_MODULE_V1_PADDING
};

接下来看下http块解析函数，ngx_http_block函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
static char *
ngx_http_block(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    char                        *rv;
    ngx_uint_t                   mi, m, s;
    ngx_conf_t                   pcf;
    ngx_http_module_t           *module;
    ngx_http_conf_ctx_t         *ctx;
    ngx_http_core_loc_conf_t    *clcf;
    ngx_http_core_srv_conf_t   **cscfp;
    ngx_http_core_main_conf_t   *cmcf;

    // 判断是否重复配置
    if (*(ngx_http_conf_ctx_t **) conf) {
        return "is duplicate";
    }

    /* the main http context */

    // 申请保存上下文的内存
    ctx = ngx_pcalloc(cf->pool, sizeof(ngx_http_conf_ctx_t));
    if (ctx == NULL) {
        return NGX_CONF_ERROR;
    }

    *(ngx_http_conf_ctx_t **) conf = ctx;


    /* count the number of the http modules and set up their indices */

    ngx_http_max_module = ngx_count_modules(cf->cycle, NGX_HTTP_MODULE);


    /* the http main_conf context, it is the same in the all http contexts */

    ctx->main_conf = ngx_pcalloc(cf->pool,
                                 sizeof(void *) * ngx_http_max_module);
    if (ctx->main_conf == NULL) {
        return NGX_CONF_ERROR;
    }


    /*
     * the http null srv_conf context, it is used to merge
     * the server{}s' srv_conf's
     */

    ctx->srv_conf = ngx_pcalloc(cf->pool, sizeof(void *) * ngx_http_max_module);
    if (ctx->srv_conf == NULL) {
        return NGX_CONF_ERROR;
    }


    /*
     * the http null loc_conf context, it is used to merge
     * the server{}s' loc_conf's
     */

    ctx->loc_conf = ngx_pcalloc(cf->pool, sizeof(void *) * ngx_http_max_module);
    if (ctx->loc_conf == NULL) {
        return NGX_CONF_ERROR;
    }


    /*
     * create the main_conf's, the null srv_conf's, and the null loc_conf's
     * of the all http modules
     */

    for (m = 0; cf->cycle->modules[m]; m++) {
        // 跳过非http模块
        if (cf->cycle->modules[m]->type != NGX_HTTP_MODULE) {
            continue;
        }

        module = cf->cycle->modules[m]->ctx;
        mi = cf->cycle->modules[m]->ctx_index;

        // 调用每个模块创建main级别conf的函数指针
        if (module->create_main_conf) {
            ctx->main_conf[mi] = module->create_main_conf(cf);
            if (ctx->main_conf[mi] == NULL) {
                return NGX_CONF_ERROR;
            }
        }

        // 调用每个模块创建server级别conf的函数指针
        if (module->create_srv_conf) {
            ctx->srv_conf[mi] = module->create_srv_conf(cf);
            if (ctx->srv_conf[mi] == NULL) {
                return NGX_CONF_ERROR;
            }
        }

        // 调用每个模块创建location级别conf的函数指针
        if (module->create_loc_conf) {
            ctx->loc_conf[mi] = module->create_loc_conf(cf);
            if (ctx->loc_conf[mi] == NULL) {
                return NGX_CONF_ERROR;
            }
        }
    }

    // 保存cf，在解析完配置后会还原
    pcf = *cf;
    // 保存http级别conf
    cf->ctx = ctx;

    for (m = 0; cf->cycle->modules[m]; m++) {
        // 跳过非http模块
        if (cf->cycle->modules[m]->type != NGX_HTTP_MODULE) {
            continue;
        }

        module = cf->cycle->modules[m]->ctx;
        //调用每个模块preconfiguration函数指针，在解析配置前需要做的
        if (module->preconfiguration) {
            if (module->preconfiguration(cf) != NGX_OK) {
                return NGX_CONF_ERROR;
            }
        }
    }

    /* parse inside the http{} block */

    cf->module_type = NGX_HTTP_MODULE;
    cf->cmd_type = NGX_HTTP_MAIN_CONF;
    // 开始解析http块里面的配置
    rv = ngx_conf_parse(cf, NULL);

    if (rv != NGX_CONF_OK) {
        goto failed;
    }

    /*
     * init http{} main_conf's, merge the server{}s' srv_conf's
     * and its location{}s' loc_conf's
     */

    // 取http main级别的conf
    cmcf = ctx->main_conf[ngx_http_core_module.ctx_index];
    cscfp = cmcf->servers.elts;

    // 调用每个http模块初始化main级别配置的函数
    for (m = 0; cf->cycle->modules[m]; m++) {
        // 跳过非http模块
        if (cf->cycle->modules[m]->type != NGX_HTTP_MODULE) {
            continue;
        }

        module = cf->cycle->modules[m]->ctx;
        mi = cf->cycle->modules[m]->ctx_index;

        /* init http{} main_conf's */
        // 调用每个模块init_main_conf函数指针
        if (module->init_main_conf) {
            rv = module->init_main_conf(cf, ctx->main_conf[mi]);
            if (rv != NGX_CONF_OK) {
                goto failed;
            }
        }

        // 合并server级别及location级别配置
        rv = ngx_http_merge_servers(cf, cmcf, module, mi);
        if (rv != NGX_CONF_OK) {
            goto failed;
        }
    }


    /* create location trees */
    // 构造http请求location匹配树
    for (s = 0; s < cmcf->servers.nelts; s++) {

        clcf = cscfp[s]->ctx->loc_conf[ngx_http_core_module.ctx_index];

        if (ngx_http_init_locations(cf, cscfp[s], clcf) != NGX_OK) {
            return NGX_CONF_ERROR;
        }

        // 构造静态location匹配树
        if (ngx_http_init_static_location_trees(cf, clcf) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }


    // 初始化各个阶段的array
    if (ngx_http_init_phases(cf, cmcf) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    // 初始化http请求头headers
    if (ngx_http_init_headers_in_hash(cf, cmcf) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    // 处理每个http模块postconfiguration
    for (m = 0; cf->cycle->modules[m]; m++) {
        if (cf->cycle->modules[m]->type != NGX_HTTP_MODULE) {
            continue;
        }

        module = cf->cycle->modules[m]->ctx;
        //调用每个模块postconfiguration函数指针，用于处理配置解析完成后的操作
        if (module->postconfiguration) {
            if (module->postconfiguration(cf) != NGX_OK) {
                return NGX_CONF_ERROR;
            }
        }
    }

    // 初始化http变量
    if (ngx_http_variables_init_vars(cf) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    /*
     * http{}'s cf->ctx was needed while the configuration merging
     * and in postconfiguration process
     */
    // 恢复cf
    *cf = pcf;

    // 初始化http各个阶段handler
    if (ngx_http_init_phase_handlers(cf, cmcf) != NGX_OK) {
        return NGX_CONF_ERROR;
    }


    /* optimize the lists of ports, addresses and server names */
    // 处理相同端口以及server_name
    if (ngx_http_optimize_servers(cf, cmcf, cmcf->ports) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    return NGX_CONF_OK;

failed:
    // 恢复cf
    *cf = pcf;
    // 返回结果
    return rv;
}

接下来分析ngx_http_optimize_servers函数，此函数用于处理IP+Port相同，不同server_name的情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
static ngx_int_t
ngx_http_optimize_servers(ngx_conf_t *cf, ngx_http_core_main_conf_t *cmcf,
    ngx_array_t *ports)
{
    ngx_uint_t             p, a;
    ngx_http_conf_port_t  *port;
    ngx_http_conf_addr_t  *addr;

    // 如果没有配置端口，则返回
    if (ports == NULL) {
        return NGX_OK;
    }

    // 遍历已经配置的port列表，
    port = ports->elts;
    for (p = 0; p < ports->nelts; p++) {

        // 将同一个port下挂的addr进行排序
        ngx_sort(port[p].addrs.elts, (size_t) port[p].addrs.nelts,
                 sizeof(ngx_http_conf_addr_t), ngx_http_cmp_conf_addrs);

        /*
         * check whether all name-based servers have the same
         * configuration as a default server for given address:port
         */

        // 如果一个addr下有多个server块，则处理server_name
        addr = port[p].addrs.elts;
        for (a = 0; a < port[p].addrs.nelts; a++) {

            if (addr[a].servers.nelts > 1
#if (NGX_PCRE)
                || addr[a].default_server->captures
#endif
               )
            {
                if (ngx_http_server_names(cf, cmcf, &addr[a]) != NGX_OK) {
                    return NGX_ERROR;
                }
            }
        }

        // 处理listen，从cycle->listening分配元素
        if (ngx_http_init_listening(cf, &port[p]) != NGX_OK) {
            return NGX_ERROR;
        }
    }

    return NGX_OK;
}

  1. 接下来分析ngx_http_server_names函数，这个函数的功能是server_name进行hash处理
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
static ngx_int_t
ngx_http_server_names(ngx_conf_t *cf, ngx_http_core_main_conf_t *cmcf,
    ngx_http_conf_addr_t *addr)
{
    ngx_int_t                   rc;
    ngx_uint_t                  n, s;
    ngx_hash_init_t             hash;
    ngx_hash_keys_arrays_t      ha;
    ngx_http_server_name_t     *name;
    ngx_http_core_srv_conf_t  **cscfp;
#if (NGX_PCRE)
    ngx_uint_t                  regex, i;

    regex = 0;
#endif

    // 初始化hash key数组
    ngx_memzero(&ha, sizeof(ngx_hash_keys_arrays_t));

    // 创建初始化hash table所需的内存池
    ha.temp_pool = ngx_create_pool(NGX_DEFAULT_POOL_SIZE, cf->log);
    if (ha.temp_pool == NULL) {
        return NGX_ERROR;
    }

    ha.pool = cf->pool;

    // 初始化hash key
    if (ngx_hash_keys_array_init(&ha, NGX_HASH_LARGE) != NGX_OK) {
        goto failed;
    }

    cscfp = addr->servers.elts;

    // 遍历此地址下的所有server块
    for (s = 0; s < addr->servers.nelts; s++) {

        name = cscfp[s]->server_names.elts;

        for (n = 0; n < cscfp[s]->server_names.nelts; n++) {

#if (NGX_PCRE)
            if (name[n].regex) {
                regex++;
                continue;
            }
#endif

            // 将server_name添加到key中
            rc = ngx_hash_add_key(&ha, &name[n].name, name[n].server,
                                  NGX_HASH_WILDCARD_KEY);

            if (rc == NGX_ERROR) {
                return NGX_ERROR;
            }

            if (rc == NGX_DECLINED) {
                ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                              "invalid server name or wildcard \"%V\" on %V",
                              &name[n].name, &addr->opt.addr_text);
                return NGX_ERROR;
            }

            // server_name出现冲突
            if (rc == NGX_BUSY) {
                ngx_log_error(NGX_LOG_WARN, cf->log, 0,
                              "conflicting server name \"%V\" on %V, ignored",
                              &name[n].name, &addr->opt.addr_text);
            }
        }
    }

    // 对hash表赋初值
    hash.key = ngx_hash_key_lc;
    hash.max_size = cmcf->server_names_hash_max_size;
    hash.bucket_size = cmcf->server_names_hash_bucket_size;
    hash.name = "server_names_hash";
    hash.pool = cf->pool;

    if (ha.keys.nelts) {
        hash.hash = &addr->hash;
        hash.temp_pool = NULL;

        if (ngx_hash_init(&hash, ha.keys.elts, ha.keys.nelts) != NGX_OK) {
            goto failed;
        }
    }

    // 前缀匹配
    if (ha.dns_wc_head.nelts) {

        ngx_qsort(ha.dns_wc_head.elts, (size_t) ha.dns_wc_head.nelts,
                  sizeof(ngx_hash_key_t), ngx_http_cmp_dns_wildcards);

        hash.hash = NULL;
        hash.temp_pool = ha.temp_pool;

        if (ngx_hash_wildcard_init(&hash, ha.dns_wc_head.elts,
                                   ha.dns_wc_head.nelts)
            != NGX_OK)
        {
            goto failed;
        }

        addr->wc_head = (ngx_hash_wildcard_t *) hash.hash;
    }

    // 后缀匹配
    if (ha.dns_wc_tail.nelts) {

        ngx_qsort(ha.dns_wc_tail.elts, (size_t) ha.dns_wc_tail.nelts,
                  sizeof(ngx_hash_key_t), ngx_http_cmp_dns_wildcards);

        hash.hash = NULL;
        hash.temp_pool = ha.temp_pool;

        if (ngx_hash_wildcard_init(&hash, ha.dns_wc_tail.elts,
                                   ha.dns_wc_tail.nelts)
            != NGX_OK)
        {
            goto failed;
        }

        addr->wc_tail = (ngx_hash_wildcard_t *) hash.hash;
    }

    ngx_destroy_pool(ha.temp_pool);

    // 处理正则
#if (NGX_PCRE)

    if (regex == 0) {
        return NGX_OK;
    }

    addr->nregex = regex;
    addr->regex = ngx_palloc(cf->pool, regex * sizeof(ngx_http_server_name_t));
    if (addr->regex == NULL) {
        return NGX_ERROR;
    }

    i = 0;

    for (s = 0; s < addr->servers.nelts; s++) {

        name = cscfp[s]->server_names.elts;
        // 遍历server_name，对regex进行处理
        for (n = 0; n < cscfp[s]->server_names.nelts; n++) {
            if (name[n].regex) {
                addr->regex[i++] = name[n];
            }
        }
    }

#endif

    return NGX_OK;

failed:
    // 释放temp_pool
    ngx_destroy_pool(ha.temp_pool);

    return NGX_ERROR;
}

  1. 接下来分析ngx_http_init_listening函数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
static ngx_int_t
ngx_http_init_listening(ngx_conf_t *cf, ngx_http_conf_port_t *port)
{
    ngx_uint_t                 i, last, bind_wildcard;
    ngx_listening_t           *ls;
    ngx_http_port_t           *hport;
    ngx_http_conf_addr_t      *addr;

    addr = port->addrs.elts;
    last = port->addrs.nelts;

    /*
     * If there is a binding to an "*:port" then we need to bind() to
     * the "*:port" only and ignore other implicit bindings.  The bindings
     * have been already sorted: explicit bindings are on the start, then
     * implicit bindings go, and wildcard binding is in the end.
     */

    if (addr[last - 1].opt.wildcard) {
        addr[last - 1].opt.bind = 1;
        bind_wildcard = 1;

    } else {
        bind_wildcard = 0;
    }

    i = 0;

    while (i < last) {

        if (bind_wildcard && !addr[i].opt.bind) {
            i++;
            continue;
        }

        // 从ngx_cycle->listening中push一个ls，跟socket相关的都保存在这里面
        ls = ngx_http_add_listening(cf, &addr[i]);
        if (ls == NULL) {
            return NGX_ERROR;
        }

        // 分配保存地址的内存，在ngx_init_connection中使用
        hport = ngx_pcalloc(cf->pool, sizeof(ngx_http_port_t));
        if (hport == NULL) {
            return NGX_ERROR;
        }

        ls->servers = hport;

        hport->naddrs = i + 1;

        switch (ls->sockaddr->sa_family) {

#if (NGX_HAVE_INET6)
        case AF_INET6:
            if (ngx_http_add_addrs6(cf, hport, addr) != NGX_OK) {
                return NGX_ERROR;
            }
            break;
#endif
        default: /* AF_INET */
            // 将addr中的内容复制到hport中
            if (ngx_http_add_addrs(cf, hport, addr) != NGX_OK) {
                return NGX_ERROR;
            }
            break;
        }

        addr++;
        last--;
    }

    return NGX_OK;
}

  1. 接下来看看ngx_http_add_addrs函数的实现，ngx_http_add_addrs6类似
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
static ngx_int_t
ngx_http_add_addrs(ngx_conf_t *cf, ngx_http_port_t *hport,
    ngx_http_conf_addr_t *addr)
{
    ngx_uint_t                 i;
    ngx_http_in_addr_t        *addrs;
    struct sockaddr_in        *sin;
    ngx_http_virtual_names_t  *vn;

    // 申请必要内存，naddrs就是外面计算的这个端口下的ip个数
    hport->addrs = ngx_pcalloc(cf->pool,
                               hport->naddrs * sizeof(ngx_http_in_addr_t));
    if (hport->addrs == NULL) {
        return NGX_ERROR;
    }

    addrs = hport->addrs;

    for (i = 0; i < hport->naddrs; i++) {

        sin = (struct sockaddr_in *) addr[i].opt.sockaddr;
        addrs[i].addr = sin->sin_addr.s_addr;
        addrs[i].conf.default_server = addr[i].default_server;
#if (NGX_HTTP_SSL)
        addrs[i].conf.ssl = addr[i].opt.ssl;
#endif
#if (NGX_HTTP_V2)
        addrs[i].conf.http2 = addr[i].opt.http2;
#endif
        addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;

        // 如果匹配server_name所需的hash桶为空，则不进行下一步操作
        if (addr[i].hash.buckets == NULL
            && (addr[i].wc_head == NULL
                || addr[i].wc_head->hash.buckets == NULL)
            && (addr[i].wc_tail == NULL
                || addr[i].wc_tail->hash.buckets == NULL)
#if (NGX_PCRE)
            && addr[i].nregex == 0
#endif
            )
        {
            continue;
        }

        // 申请保存server_name匹配所需内存
        vn = ngx_palloc(cf->pool, sizeof(ngx_http_virtual_names_t));
        if (vn == NULL) {
            return NGX_ERROR;
        }

        addrs[i].conf.virtual_names = vn;

        // 保存server_name匹配所需的hash值
        vn->names.hash = addr[i].hash;
        vn->names.wc_head = addr[i].wc_head;
        vn->names.wc_tail = addr[i].wc_tail;
#if (NGX_PCRE)
        vn->nregex = addr[i].nregex;
        vn->regex = addr[i].regex;
#endif
    }

    return NGX_OK;
}

  1. 接下来看看函数的实现
1

Nginx

Nginx的http_core模块

1、相关配置

核心模块配置如下

1
2
3
4
5
6
7
8
9

server {
    listen 80;
    location / {
        root html;
        index index.html;
    }
}

以上是http模块最核心的配置，一条listen指令，一条location指令，能够监听80端口，提供简单的web服务。

2、相关代码

  1. 首先看下解析server指令的代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
static char *
ngx_http_core_server(ngx_conf_t *cf, ngx_command_t *cmd, void *dummy)
{
    char                        *rv;
    void                        *mconf;
    size_t                       len;
    u_char                      *p;
    ngx_uint_t                   i;
    ngx_conf_t                   pcf;
    ngx_http_module_t           *module;
    struct sockaddr_in          *sin;
    ngx_http_conf_ctx_t         *ctx, *http_ctx;
    ngx_http_listen_opt_t        lsopt;
    ngx_http_core_srv_conf_t    *cscf, **cscfp;
    ngx_http_core_main_conf_t   *cmcf;

    // 首先分配上下文内存
    ctx = ngx_pcalloc(cf->pool, sizeof(ngx_http_conf_ctx_t));
    if (ctx == NULL) {
        return NGX_CONF_ERROR;
    }

    // 保存http模块main级别的conf
    http_ctx = cf->ctx;
    ctx->main_conf = http_ctx->main_conf;

    /* the server{}'s srv_conf */

    // 预分配保存server级别的内存
    ctx->srv_conf = ngx_pcalloc(cf->pool, sizeof(void *) * ngx_http_max_module);
    if (ctx->srv_conf == NULL) {
        return NGX_CONF_ERROR;
    }

    /* the server{}'s loc_conf */

    // 预分配location级别的内存
    ctx->loc_conf = ngx_pcalloc(cf->pool, sizeof(void *) * ngx_http_max_module);
    if (ctx->loc_conf == NULL) {
        return NGX_CONF_ERROR;
    }

    for (i = 0; cf->cycle->modules[i]; i++) {
        if (cf->cycle->modules[i]->type != NGX_HTTP_MODULE) {
            continue;
        }

        module = cf->cycle->modules[i]->ctx;

        // 调用每个模块create_srv_conf函数指针，创建server级别conf
        if (module->create_srv_conf) {
            mconf = module->create_srv_conf(cf);
            if (mconf == NULL) {
                return NGX_CONF_ERROR;
            }
            // 使用模块索引保存创建的server级别conf
            ctx->srv_conf[cf->cycle->modules[i]->ctx_index] = mconf;
        }

        // 调用每个模块create_loc_conf函数指针，创建location级别conf
        if (module->create_loc_conf) {
            mconf = module->create_loc_conf(cf);
            if (mconf == NULL) {
                return NGX_CONF_ERROR;
            }
            // 使用模块索引保存创建的location级别conf
            ctx->loc_conf[cf->cycle->modules[i]->ctx_index] = mconf;
        }
    }


    /* the server configuration context */

    cscf = ctx->srv_conf[ngx_http_core_module.ctx_index];
    cscf->ctx = ctx;

    //获取http的主配置
    cmcf = ctx->main_conf[ngx_http_core_module.ctx_index];

    // 创建servers数组
    cscfp = ngx_array_push(&cmcf->servers);
    if (cscfp == NULL) {
        return NGX_CONF_ERROR;
    }

    //保存cscfp
    *cscfp = cscf;


    /* parse inside server{} */

    pcf = *cf;
    cf->ctx = ctx;
    cf->cmd_type = NGX_HTTP_SRV_CONF;

    // 开始解析server块配置
    rv = ngx_conf_parse(cf, NULL);

    // 还原cf
    *cf = pcf;

    // 处理没有监听端口的情况, 默认80
    if (rv == NGX_CONF_OK && !cscf->listen) {
        ngx_memzero(&lsopt, sizeof(ngx_http_listen_opt_t));

        p = ngx_pcalloc(cf->pool, sizeof(struct sockaddr_in));
        if (p == NULL) {
            return NGX_CONF_ERROR;
        }

        lsopt.sockaddr = (struct sockaddr *) p;

        sin = (struct sockaddr_in *) p;

        sin->sin_family = AF_INET;
#if (NGX_WIN32)
        sin->sin_port = htons(80);
#else
        sin->sin_port = htons((getuid() == 0) ? 80 : 8000);
#endif
        sin->sin_addr.s_addr = INADDR_ANY;

        lsopt.socklen = sizeof(struct sockaddr_in);

        lsopt.backlog = NGX_LISTEN_BACKLOG;
        lsopt.rcvbuf = -1;
        lsopt.sndbuf = -1;
#if (NGX_HAVE_SETFIB)
        lsopt.setfib = -1;
#endif
#if (NGX_HAVE_TCP_FASTOPEN)
        lsopt.fastopen = -1;
#endif
        lsopt.wildcard = 1;

        len = NGX_INET_ADDRSTRLEN + sizeof(":65535") - 1;

        p = ngx_pnalloc(cf->pool, len);
        if (p == NULL) {
            return NGX_CONF_ERROR;
        }

        lsopt.addr_text.data = p;
        lsopt.addr_text.len = ngx_sock_ntop(lsopt.sockaddr, lsopt.socklen, p,
                                            len, 1);

        // 添加端口到listening列表
        if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }

    return rv;
}

  1. 接下来看看解析listen指令的函数, ngx_http_core_listen的实现
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
static char *
ngx_http_core_listen(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_core_srv_conf_t *cscf = conf;

    ngx_str_t              *value, size;
    ngx_url_t               u;
    ngx_uint_t              n;
    ngx_http_listen_opt_t   lsopt;

    cscf->listen = 1;

    value = cf->args->elts;

    // URL结构体内存空间置零
    ngx_memzero(&u, sizeof(ngx_url_t));

    u.url = value[1];
    u.listen = 1;
    u.default_port = 80;

    // 解析listen指令后的ip+port
    if (ngx_parse_url(cf->pool, &u) != NGX_OK) {
        if (u.err) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "%s in \"%V\" of the \"listen\" directive",
                               u.err, &u.url);
        }

        return NGX_CONF_ERROR;
    }

    ngx_memzero(&lsopt, sizeof(ngx_http_listen_opt_t));

    lsopt.backlog = NGX_LISTEN_BACKLOG;
    lsopt.rcvbuf = -1;
    lsopt.sndbuf = -1;
#if (NGX_HAVE_SETFIB)
    lsopt.setfib = -1;
#endif
#if (NGX_HAVE_TCP_FASTOPEN)
    lsopt.fastopen = -1;
#endif
#if (NGX_HAVE_INET6)
    lsopt.ipv6only = 1;
#endif
    // 开始解析listen指令后的参数，第一个参数是listen，第二个参数是IP端口，所以n从2开始
    for (n = 2; n < cf->args->nelts; n++) {

        // 解析default server参数，并设置标志位
        if (ngx_strcmp(value[n].data, "default_server") == 0
            || ngx_strcmp(value[n].data, "default") == 0)
        {
            lsopt.default_server = 1;
            continue;
        }

        // 解析bind参数，并设置标志位
        if (ngx_strcmp(value[n].data, "bind") == 0) {
            lsopt.set = 1;
            lsopt.bind = 1;
            continue;
        }

#if (NGX_HAVE_SETFIB)
        if (ngx_strncmp(value[n].data, "setfib=", 7) == 0) {
            lsopt.setfib = ngx_atoi(value[n].data + 7, value[n].len - 7);
            lsopt.set = 1;
            lsopt.bind = 1;

            if (lsopt.setfib == NGX_ERROR) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid setfib \"%V\"", &value[n]);
                return NGX_CONF_ERROR;
            }

            continue;
        }
#endif

#if (NGX_HAVE_TCP_FASTOPEN)
        // 解析fastopen参数，设置标志位和值
        if (ngx_strncmp(value[n].data, "fastopen=", 9) == 0) {
            lsopt.fastopen = ngx_atoi(value[n].data + 9, value[n].len - 9);
            lsopt.set = 1;
            lsopt.bind = 1;

            if (lsopt.fastopen == NGX_ERROR) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid fastopen \"%V\"", &value[n]);
                return NGX_CONF_ERROR;
            }

            continue;
        }
#endif

        // 解析backlog参数，设置标志位和值
        if (ngx_strncmp(value[n].data, "backlog=", 8) == 0) {
            lsopt.backlog = ngx_atoi(value[n].data + 8, value[n].len - 8);
            lsopt.set = 1;
            lsopt.bind = 1;

            if (lsopt.backlog == NGX_ERROR || lsopt.backlog == 0) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid backlog \"%V\"", &value[n]);
                return NGX_CONF_ERROR;
            }

            continue;
        }

        // recv buf参数，设置标志位和值
        if (ngx_strncmp(value[n].data, "rcvbuf=", 7) == 0) {
            size.len = value[n].len - 7;
            size.data = value[n].data + 7;

            lsopt.rcvbuf = ngx_parse_size(&size);
            lsopt.set = 1;
            lsopt.bind = 1;

            if (lsopt.rcvbuf == NGX_ERROR) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid rcvbuf \"%V\"", &value[n]);
                return NGX_CONF_ERROR;
            }

            continue;
        }

        // send buf参数，设置标志位和值
        if (ngx_strncmp(value[n].data, "sndbuf=", 7) == 0) {
            size.len = value[n].len - 7;
            size.data = value[n].data + 7;

            lsopt.sndbuf = ngx_parse_size(&size);
            lsopt.set = 1;
            lsopt.bind = 1;

            if (lsopt.sndbuf == NGX_ERROR) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid sndbuf \"%V\"", &value[n]);
                return NGX_CONF_ERROR;
            }

            continue;
        }

        // 解析accept_filter参数，设置标志位
        if (ngx_strncmp(value[n].data, "accept_filter=", 14) == 0) {
#if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER)
            lsopt.accept_filter = (char *) &value[n].data[14];
            lsopt.set = 1;
            lsopt.bind = 1;
#else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "accept filters \"%V\" are not supported "
                               "on this platform, ignored",
                               &value[n]);
#endif
            continue;
        }

        // deferred 参数
        if (ngx_strcmp(value[n].data, "deferred") == 0) {
#if (NGX_HAVE_DEFERRED_ACCEPT && defined TCP_DEFER_ACCEPT)
            lsopt.deferred_accept = 1;
            lsopt.set = 1;
            lsopt.bind = 1;
#else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "the deferred accept is not supported "
                               "on this platform, ignored");
#endif
            continue;
        }

        // 解析ipv6参数，设置标志位
        if (ngx_strncmp(value[n].data, "ipv6only=o", 10) == 0) {
#if (NGX_HAVE_INET6 && defined IPV6_V6ONLY)
            if (ngx_strcmp(&value[n].data[10], "n") == 0) {
                lsopt.ipv6only = 1;

            } else if (ngx_strcmp(&value[n].data[10], "ff") == 0) {
                lsopt.ipv6only = 0;

            } else {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid ipv6only flags \"%s\"",
                                   &value[n].data[9]);
                return NGX_CONF_ERROR;
            }

            lsopt.set = 1;
            lsopt.bind = 1;

            continue;
#else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "ipv6only is not supported "
                               "on this platform");
            return NGX_CONF_ERROR;
#endif
        }

        // reuseport 参数，设置标志位
        if (ngx_strcmp(value[n].data, "reuseport") == 0) {
#if (NGX_HAVE_REUSEPORT)
            lsopt.reuseport = 1;
            lsopt.set = 1;
            lsopt.bind = 1;
#else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "reuseport is not supported "
                               "on this platform, ignored");
#endif
            continue;
        }

        // 解析ssl参数，设置标志位
        if (ngx_strcmp(value[n].data, "ssl") == 0) {
#if (NGX_HTTP_SSL)
            lsopt.ssl = 1;
            continue;
#else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "the \"ssl\" parameter requires "
                               "ngx_http_ssl_module");
            return NGX_CONF_ERROR;
#endif
        }

        // 解析http2参数，设置标志位
        if (ngx_strcmp(value[n].data, "http2") == 0) {
#if (NGX_HTTP_V2)
            lsopt.http2 = 1;
            continue;
#else
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "the \"http2\" parameter requires "
                               "ngx_http_v2_module");
            return NGX_CONF_ERROR;
#endif
        }

        // 解析spdy参数
        if (ngx_strcmp(value[n].data, "spdy") == 0) {
            ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
                               "invalid parameter \"spdy\": "
                               "ngx_http_spdy_module was superseded "
                               "by ngx_http_v2_module");
            continue;
        }

        // 解析so_keepalive，设置标志位
        if (ngx_strncmp(value[n].data, "so_keepalive=", 13) == 0) {

            if (ngx_strcmp(&value[n].data[13], "on") == 0) {
                lsopt.so_keepalive = 1;

            } else if (ngx_strcmp(&value[n].data[13], "off") == 0) {
                lsopt.so_keepalive = 2;

            } else {

#if (NGX_HAVE_KEEPALIVE_TUNABLE)
                u_char     *p, *end;
                ngx_str_t   s;

                end = value[n].data + value[n].len;
                s.data = value[n].data + 13;

                p = ngx_strlchr(s.data, end, ':');
                if (p == NULL) {
                    p = end;
                }

                if (p > s.data) {
                    s.len = p - s.data;

                    lsopt.tcp_keepidle = ngx_parse_time(&s, 1);
                    if (lsopt.tcp_keepidle == (time_t) NGX_ERROR) {
                        goto invalid_so_keepalive;
                    }
                }

                s.data = (p < end) ? (p + 1) : end;

                p = ngx_strlchr(s.data, end, ':');
                if (p == NULL) {
                    p = end;
                }

                if (p > s.data) {
                    s.len = p - s.data;

                    lsopt.tcp_keepintvl = ngx_parse_time(&s, 1);
                    if (lsopt.tcp_keepintvl == (time_t) NGX_ERROR) {
                        goto invalid_so_keepalive;
                    }
                }

                s.data = (p < end) ? (p + 1) : end;

                if (s.data < end) {
                    s.len = end - s.data;

                    lsopt.tcp_keepcnt = ngx_atoi(s.data, s.len);
                    if (lsopt.tcp_keepcnt == NGX_ERROR) {
                        goto invalid_so_keepalive;
                    }
                }

                if (lsopt.tcp_keepidle == 0 && lsopt.tcp_keepintvl == 0
                    && lsopt.tcp_keepcnt == 0)
                {
                    goto invalid_so_keepalive;
                }

                lsopt.so_keepalive = 1;

#else

                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "the \"so_keepalive\" parameter accepts "
                                   "only \"on\" or \"off\" on this platform");
                return NGX_CONF_ERROR;

#endif
            }

            lsopt.set = 1;
            lsopt.bind = 1;

            continue;

#if (NGX_HAVE_KEEPALIVE_TUNABLE)
        invalid_so_keepalive:

            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid so_keepalive value: \"%s\"",
                               &value[n].data[13]);
            return NGX_CONF_ERROR;
#endif
        }

        // 解析pp协议参数，开启pp协议标志位
        if (ngx_strcmp(value[n].data, "proxy_protocol") == 0) {
            lsopt.proxy_protocol = 1;
            continue;
        }

        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "invalid parameter \"%V\"", &value[n]);
        return NGX_CONF_ERROR;
    }

    // 处理监听端口
    for (n = 0; n < u.naddrs; n++) {
        lsopt.sockaddr = u.addrs[n].sockaddr;
        lsopt.socklen = u.addrs[n].socklen;
        lsopt.addr_text = u.addrs[n].name;
        // 获取通配符关键字
        lsopt.wildcard = ngx_inet_wildcard(lsopt.sockaddr);

        // 将端口放入全局端口列表中
        if (ngx_http_add_listen(cf, cscf, &lsopt) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }

    return NGX_CONF_OK;
}



  1. 接下来看看解析location指令的函数, ngx_http_core_location的实现
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
static char *
ngx_http_core_location(ngx_conf_t *cf, ngx_command_t *cmd, void *dummy)
{
    char                      *rv;
    u_char                    *mod;
    size_t                     len;
    ngx_str_t                 *value, *name;
    ngx_uint_t                 i;
    ngx_conf_t                 save;
    ngx_http_module_t         *module;
    ngx_http_conf_ctx_t       *ctx, *pctx;
    ngx_http_core_loc_conf_t  *clcf, *pclcf;

    // 首先申请保存配置的内存
    ctx = ngx_pcalloc(cf->pool, sizeof(ngx_http_conf_ctx_t));
    if (ctx == NULL) {
        return NGX_CONF_ERROR;
    }

    // 设置main级别和server级别配置
    pctx = cf->ctx;
    ctx->main_conf = pctx->main_conf;
    ctx->srv_conf = pctx->srv_conf;

    ctx->loc_conf = ngx_pcalloc(cf->pool, sizeof(void *) * ngx_http_max_module);
    if (ctx->loc_conf == NULL) {
        return NGX_CONF_ERROR;
    }

    // 遍历http模块，调用创建location级别配置函数
    for (i = 0; cf->cycle->modules[i]; i++) {
        // 跳过非http模块
        if (cf->cycle->modules[i]->type != NGX_HTTP_MODULE) {
            continue;
        }

        module = cf->cycle->modules[i]->ctx;

        if (module->create_loc_conf) {
            ctx->loc_conf[cf->cycle->modules[i]->ctx_index] =
                                                   module->create_loc_conf(cf);
            if (ctx->loc_conf[cf->cycle->modules[i]->ctx_index] == NULL) {
                return NGX_CONF_ERROR;
            }
        }
    }

    // 取出core模块的location级别配置
    clcf = ctx->loc_conf[ngx_http_core_module.ctx_index];
    clcf->loc_conf = ctx->loc_conf;

    value = cf->args->elts;

    // 处理location指令后的参数，比如/ ~ = 以及其他正则表达式
    if (cf->args->nelts == 3) {

        len = value[1].len;
        mod = value[1].data;
        name = &value[2];

        if (len == 1 && mod[0] == '=') {

            clcf->name = *name;
            clcf->exact_match = 1;

        } else if (len == 2 && mod[0] == '^' && mod[1] == '~') {

            clcf->name = *name;
            clcf->noregex = 1;

        } else if (len == 1 && mod[0] == '~') {

            if (ngx_http_core_regex_location(cf, clcf, name, 0) != NGX_OK) {
                return NGX_CONF_ERROR;
            }

        } else if (len == 2 && mod[0] == '~' && mod[1] == '*') {

            if (ngx_http_core_regex_location(cf, clcf, name, 1) != NGX_OK) {
                return NGX_CONF_ERROR;
            }

        } else {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid location modifier \"%V\"", &value[1]);
            return NGX_CONF_ERROR;
        }

    } else {

        name = &value[1];

        if (name->data[0] == '=') {

            clcf->name.len = name->len - 1;
            clcf->name.data = name->data + 1;
            clcf->exact_match = 1;

        } else if (name->data[0] == '^' && name->data[1] == '~') {

            clcf->name.len = name->len - 2;
            clcf->name.data = name->data + 2;
            clcf->noregex = 1;

        } else if (name->data[0] == '~') {

            name->len--;
            name->data++;

            if (name->data[0] == '*') {

                name->len--;
                name->data++;

                // 处理正则表达式
                if (ngx_http_core_regex_location(cf, clcf, name, 1) != NGX_OK) {
                    return NGX_CONF_ERROR;
                }

            } else {
                if (ngx_http_core_regex_location(cf, clcf, name, 0) != NGX_OK) {
                    return NGX_CONF_ERROR;
                }
            }

        } else {

            // 保存clcf的name
            clcf->name = *name;

            if (name->data[0] == '@') {
                clcf->named = 1;
            }
        }
    }

    // 设置location级别的配置
    pclcf = pctx->loc_conf[ngx_http_core_module.ctx_index];

    // 判断是否微location类型配置
    if (cf->cmd_type == NGX_HTTP_LOC_CONF) {

        /* nested location */

#if 0
        clcf->prev_location = pclcf;
#endif

        if (pclcf->exact_match) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "location \"%V\" cannot be inside "
                               "the exact location \"%V\"",
                               &clcf->name, &pclcf->name);
            return NGX_CONF_ERROR;
        }

        if (pclcf->named) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "location \"%V\" cannot be inside "
                               "the named location \"%V\"",
                               &clcf->name, &pclcf->name);
            return NGX_CONF_ERROR;
        }

        if (clcf->named) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "named location \"%V\" can be "
                               "on the server level only",
                               &clcf->name);
            return NGX_CONF_ERROR;
        }

        len = pclcf->name.len;

#if (NGX_PCRE)
        if (clcf->regex == NULL
            && ngx_filename_cmp(clcf->name.data, pclcf->name.data, len) != 0)
#else
        if (ngx_filename_cmp(clcf->name.data, pclcf->name.data, len) != 0)
#endif
        {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "location \"%V\" is outside location \"%V\"",
                               &clcf->name, &pclcf->name);
            return NGX_CONF_ERROR;
        }
    }

    // 添加location
    if (ngx_http_add_location(cf, &pclcf->locations, clcf) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    // 保持cf结构体
    save = *cf;
    cf->ctx = ctx;
    cf->cmd_type = NGX_HTTP_LOC_CONF;

    // 解析location里面的指令
    rv = ngx_conf_parse(cf, NULL);

    // 恢复原来的配置
    *cf = save;

    return rv;
}

  1. 接下来看看root指令解析函数，ngx_http_core_root的实现
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

static char *
ngx_http_core_root(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_core_loc_conf_t *clcf = conf;

    ngx_str_t                  *value;
    ngx_int_t                   alias;
    ngx_uint_t                  n;
    ngx_http_script_compile_t   sc;

    alias = (cmd->name.len == sizeof("alias") - 1) ? 1 : 0;

    if (clcf->root.data) {

        if ((clcf->alias != 0) == alias) {
            return "is duplicate";
        }

        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "\"%V\" directive is duplicate, "
                           "\"%s\" directive was specified earlier",
                           &cmd->name, clcf->alias ? "alias" : "root");

        return NGX_CONF_ERROR;
    }

    if (clcf->named && alias) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "the \"alias\" directive cannot be used "
                           "inside the named location");

        return NGX_CONF_ERROR;
    }

    value = cf->args->elts;

    // 处理第一个参数
    if (ngx_strstr(value[1].data, "$document_root")
        || ngx_strstr(value[1].data, "${document_root}"))
    {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "the $document_root variable cannot be used "
                           "in the \"%V\" directive",
                           &cmd->name);

        return NGX_CONF_ERROR;
    }

    if (ngx_strstr(value[1].data, "$realpath_root")
        || ngx_strstr(value[1].data, "${realpath_root}"))
    {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "the $realpath_root variable cannot be used "
                           "in the \"%V\" directive",
                           &cmd->name);

        return NGX_CONF_ERROR;
    }

    clcf->alias = alias ? clcf->name.len : 0;
    clcf->root = value[1];

    if (!alias && clcf->root.len > 0
        && clcf->root.data[clcf->root.len - 1] == '/')
    {
        clcf->root.len--;
    }

    if (clcf->root.data[0] != '$') {
        if (ngx_conf_full_name(cf->cycle, &clcf->root, 0) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }

    // 统计变量数量
    n = ngx_http_script_variables_count(&clcf->root);

    // 结构体置空
    ngx_memzero(&sc, sizeof(ngx_http_script_compile_t));
    sc.variables = n;

#if (NGX_PCRE)
    if (alias && clcf->regex) {
        clcf->alias = NGX_MAX_SIZE_T_VALUE;
        n = 1;
    }
#endif

    //处理变量
    if (n) {
        sc.cf = cf;
        sc.source = &clcf->root;
        sc.lengths = &clcf->root_lengths;
        sc.values = &clcf->root_values;
        sc.complete_lengths = 1;
        sc.complete_values = 1;

        if (ngx_http_script_compile(&sc) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }

    return NGX_CONF_OK;
}

  1. 接下来看看type指令解析函数, ngx_http_core_type的实现
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
static char *
ngx_http_core_type(ngx_conf_t *cf, ngx_command_t *dummy, void *conf)
{
    ngx_http_core_loc_conf_t *clcf = conf;

    ngx_str_t       *value, *content_type, *old;
    ngx_uint_t       i, n, hash;
    ngx_hash_key_t  *type;

    value = cf->args->elts;
    // 解析第一个参数，判断是否为include
    if (ngx_strcmp(value[0].data, "include") == 0) {
        if (cf->args->nelts != 2) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid number of arguments"
                               " in \"include\" directive");
            return NGX_CONF_ERROR;
        }

        return ngx_conf_include(cf, dummy, conf);
    }

    //申请保存content_type的内存
    content_type = ngx_palloc(cf->pool, sizeof(ngx_str_t));
    if (content_type == NULL) {
        return NGX_CONF_ERROR;
    }

    // 取content_type的值
    *content_type = value[0];

    for (i = 1; i < cf->args->nelts; i++) {

        hash = ngx_hash_strlow(value[i].data, value[i].data, value[i].len);

        type = clcf->types->elts;
        for (n = 0; n < clcf->types->nelts; n++) {
            if (ngx_strcmp(value[i].data, type[n].key.data) == 0) {
                old = type[n].value;
                type[n].value = content_type;

                ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
                                   "duplicate extension \"%V\", "
                                   "content type: \"%V\", "
                                   "previous content type: \"%V\"",
                                   &value[i], content_type, old);
                goto next;
            }
        }

        // 从types中取出一个元素
        type = ngx_array_push(clcf->types);
        if (type == NULL) {
            return NGX_CONF_ERROR;
        }

        //保存
        type->key = value[i];
        type->key_hash = hash;
        type->value = content_type;

    next:
        continue;
    }

    return NGX_CONF_OK;
}

  1. 接下来看看server_name指令的解析函数, ngx_http_core_server_name的实现
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
static char *
ngx_http_core_server_name(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_core_srv_conf_t *cscf = conf;

    u_char                   ch;
    ngx_str_t               *value;
    ngx_uint_t               i;
    ngx_http_server_name_t  *sn;

    // 取参数
    value = cf->args->elts;

    // 遍历参数个数
    for (i = 1; i < cf->args->nelts; i++) {

        ch = value[i].data[0];

        // 判断是否为通配符
        if ((ch == '*' && (value[i].len < 3 || value[i].data[1] != '.'))
            || (ch == '.' && value[i].len < 2))
        {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "server name \"%V\" is invalid", &value[i]);
            return NGX_CONF_ERROR;
        }

        // 判断server_name中是否含有/
        if (ngx_strchr(value[i].data, '/')) {
            ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
                               "server name \"%V\" has suspicious symbols",
                               &value[i]);
        }

        // 取一个server_name的元素
        sn = ngx_array_push(&cscf->server_names);
        if (sn == NULL) {
            return NGX_CONF_ERROR;
        }

#if (NGX_PCRE)
        sn->regex = NULL;
#endif
        // 保存server块配置结构体
        sn->server = cscf;

        // 判断server_name参数是否为主机名称，如果是，则使用cycle->hostname，否则server_name的值就是value的值
        if (ngx_strcasecmp(value[i].data, (u_char *) "$hostname") == 0) {
            sn->name = cf->cycle->hostname;

        } else {
            sn->name = value[i];
        }

        // 处理server_name转小写
        if (value[i].data[0] != '~') {
            ngx_strlow(sn->name.data, sn->name.data, sn->name.len);
            continue;
        }

#if (NGX_PCRE)
        // 处理正则表达式
        {
        u_char               *p;
        ngx_regex_compile_t   rc;
        u_char                errstr[NGX_MAX_CONF_ERRSTR];

        if (value[i].len == 1) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "empty regex in server name \"%V\"", &value[i]);
            return NGX_CONF_ERROR;
        }

        // 跳过第一个符号
        value[i].len--;
        value[i].data++;

        ngx_memzero(&rc, sizeof(ngx_regex_compile_t));

        rc.pattern = value[i];
        rc.err.len = NGX_MAX_CONF_ERRSTR;
        rc.err.data = errstr;

        for (p = value[i].data; p < value[i].data + value[i].len; p++) {
            if (*p >= 'A' && *p <= 'Z') {
                rc.options = NGX_REGEX_CASELESS;
                break;
            }
        }

        // 编译正则表达式
        sn->regex = ngx_http_regex_compile(cf, &rc);
        if (sn->regex == NULL) {
            return NGX_CONF_ERROR;
        }

        sn->name = value[i];
        cscf->captures = (rc.captures > 0);
        }
#else
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "using regex \"%V\" "
                           "requires PCRE library", &value[i]);

        return NGX_CONF_ERROR;
#endif
    }

    return NGX_CONF_OK;
}

  1. 接下来看看error_page指令的解析函数, ngx_http_core_error_page的实现
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
ngx_http_core_error_page(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_core_loc_conf_t *clcf = conf;

    u_char                            *p;
    ngx_int_t                          overwrite;
    ngx_str_t                         *value, uri, args;
    ngx_uint_t                         i, n;
    ngx_http_err_page_t               *err;
    ngx_http_complex_value_t           cv;
    ngx_http_compile_complex_value_t   ccv;

    if (clcf->error_pages == NULL) {
        clcf->error_pages = ngx_array_create(cf->pool, 4,
                                             sizeof(ngx_http_err_page_t));
        if (clcf->error_pages == NULL) {
            return NGX_CONF_ERROR;
        }
    }

    value = cf->args->elts;

    i = cf->args->nelts - 2;

    // 解析重定向的code
    if (value[i].data[0] == '=') {
        if (i == 1) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid value \"%V\"", &value[i]);
            return NGX_CONF_ERROR;
        }

        if (value[i].len > 1) {
            overwrite = ngx_atoi(&value[i].data[1], value[i].len - 1);

            if (overwrite == NGX_ERROR) {
                ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                   "invalid value \"%V\"", &value[i]);
                return NGX_CONF_ERROR;
            }

        } else {
            overwrite = 0;
        }

        n = 2;

    } else {
        overwrite = -1;
        n = 1;
    }

    uri = value[cf->args->nelts - 1];

    ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));

    ccv.cf = cf;
    ccv.value = &uri;
    ccv.complex_value = &cv;
    // 解析uri变量
    if (ngx_http_compile_complex_value(&ccv) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

    ngx_str_null(&args);
    // 解析uri及参数
    if (cv.lengths == NULL && uri.len && uri.data[0] == '/') {
        //搜索查询参数起点
        p = (u_char *) ngx_strchr(uri.data, '?');

        if (p) {
            cv.value.len = p - uri.data;
            cv.value.data = uri.data;
            p++;
            args.len = (uri.data + uri.len) - p;
            args.data = p;
        }
    }

    // 开始从第一个参数解析需要重定向的code
    for (i = 1; i < cf->args->nelts - n; i++) {
        err = ngx_array_push(clcf->error_pages);
        if (err == NULL) {
            return NGX_CONF_ERROR;
        }
        // 解析数字code
        err->status = ngx_atoi(value[i].data, value[i].len);
        // 如果解析出错或者是499，则直接返回错误
        if (err->status == NGX_ERROR || err->status == 499) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "invalid value \"%V\"", &value[i]);
            return NGX_CONF_ERROR;
        }
        // status范围300-599之间
        if (err->status < 300 || err->status > 599) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "value \"%V\" must be between 300 and 599",
                               &value[i]);
            return NGX_CONF_ERROR;
        }

        err->overwrite = overwrite;

        if (overwrite == -1) {
            switch (err->status) {
                case NGX_HTTP_TO_HTTPS:
                case NGX_HTTPS_CERT_ERROR:
                case NGX_HTTPS_NO_CERT:
                case NGX_HTTP_REQUEST_HEADER_TOO_LARGE:
                    err->overwrite = NGX_HTTP_BAD_REQUEST;
            }
        }

        err->value = cv;
        err->args = args;
    }

    return NGX_CONF_OK;
}

error_page主要用于内部重定向，在special response的时候处理相关逻辑

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
ngx_int_t
ngx_http_special_response_handler(ngx_http_request_t *r, ngx_int_t error)
{
    ngx_uint_t                 i, err;
    ngx_http_err_page_t       *err_page;
    ngx_http_core_loc_conf_t  *clcf;

    ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "http special response: %i, \"%V?%V\"",
                   error, &r->uri, &r->args);

    r->err_status = error;

    if (r->keepalive) {
        switch (error) {
            case NGX_HTTP_BAD_REQUEST:
            case NGX_HTTP_REQUEST_ENTITY_TOO_LARGE:
            case NGX_HTTP_REQUEST_URI_TOO_LARGE:
            case NGX_HTTP_TO_HTTPS:
            case NGX_HTTPS_CERT_ERROR:
            case NGX_HTTPS_NO_CERT:
            case NGX_HTTP_INTERNAL_SERVER_ERROR:
            case NGX_HTTP_NOT_IMPLEMENTED:
                r->keepalive = 0;
        }
    }

    if (r->lingering_close) {
        switch (error) {
            case NGX_HTTP_BAD_REQUEST:
            case NGX_HTTP_TO_HTTPS:
            case NGX_HTTPS_CERT_ERROR:
            case NGX_HTTPS_NO_CERT:
                r->lingering_close = 0;
        }
    }
    // content-length置为0
    r->headers_out.content_type.len = 0;

    clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
    // 这里对error_page以及uri_changes判断
    if (!r->error_page && clcf->error_pages && r->uri_changes != 0) {
        
        if (clcf->recursive_error_pages == 0) {
            r->error_page = 1;
        }

        err_page = clcf->error_pages->elts;
        // 找到对应code
        for (i = 0; i < clcf->error_pages->nelts; i++) {
            if (err_page[i].status == error) {
                // 直接调用error_page逻辑
                return ngx_http_send_error_page(r, &err_page[i]);
            }
        }
    }

    r->expect_tested = 1;

    if (ngx_http_discard_request_body(r) != NGX_OK) {
        r->keepalive = 0;
    }

    if (clcf->msie_refresh
        && r->headers_in.msie
        && (error == NGX_HTTP_MOVED_PERMANENTLY
            || error == NGX_HTTP_MOVED_TEMPORARILY))
    {
        return ngx_http_send_refresh(r);
    }

    if (error == NGX_HTTP_CREATED) {
        /* 201 */
        err = 0;

    } else if (error == NGX_HTTP_NO_CONTENT) {
        /* 204 */
        err = 0;

    } else if (error >= NGX_HTTP_MOVED_PERMANENTLY
               && error < NGX_HTTP_LAST_3XX)
    {
        /* 3XX */
        err = error - NGX_HTTP_MOVED_PERMANENTLY + NGX_HTTP_OFF_3XX;

    } else if (error >= NGX_HTTP_BAD_REQUEST
               && error < NGX_HTTP_LAST_4XX)
    {
        /* 4XX */
        err = error - NGX_HTTP_BAD_REQUEST + NGX_HTTP_OFF_4XX;

    } else if (error >= NGX_HTTP_NGINX_CODES
               && error < NGX_HTTP_LAST_5XX)
    {
        /* 49X, 5XX */
        err = error - NGX_HTTP_NGINX_CODES + NGX_HTTP_OFF_5XX;
        switch (error) {
            case NGX_HTTP_TO_HTTPS:
            case NGX_HTTPS_CERT_ERROR:
            case NGX_HTTPS_NO_CERT:
            case NGX_HTTP_REQUEST_HEADER_TOO_LARGE:
                r->err_status = NGX_HTTP_BAD_REQUEST;
        }

    } else {
        /* unknown code, zero body */
        err = 0;
    }

    return ngx_http_send_special_response(r, clcf, err);
}

接下来看下ngx_http_send_error_page的函数实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
static ngx_int_t
ngx_http_send_error_page(ngx_http_request_t *r, ngx_http_err_page_t *err_page)
{
    ngx_int_t                  overwrite;
    ngx_str_t                  uri, args;
    ngx_table_elt_t           *location;
    ngx_http_core_loc_conf_t  *clcf;

    overwrite = err_page->overwrite;

    if (overwrite && overwrite != NGX_HTTP_OK) {
        r->expect_tested = 1;
    }

    // 如果配置了code，则将error_code置为配置的code
    if (overwrite >= 0) {
        r->err_status = overwrite;
    }

    // 处理url中的变量
    if (ngx_http_complex_value(r, &err_page->value, &uri) != NGX_OK) {
        return NGX_ERROR;
    }

    // 如果url是/开始
    if (uri.len && uri.data[0] == '/') {

        if (err_page->value.lengths) {
            ngx_http_split_args(r, &uri, &args);

        } else {
            args = err_page->args;
        }

        if (r->method != NGX_HTTP_HEAD) {
            r->method = NGX_HTTP_GET;
            r->method_name = ngx_http_core_get_method;
        }

        return ngx_http_internal_redirect(r, &uri, &args);
    }

    // 跳转到name的location
    if (uri.len && uri.data[0] == '@') {
        return ngx_http_named_location(r, &uri);
    }

    r->expect_tested = 1;

    if (ngx_http_discard_request_body(r) != NGX_OK) {
        r->keepalive = 0;
    }

    // 从headers中取一个元素，用作重定向
    location = ngx_list_push(&r->headers_out.headers);

    if (location == NULL) {
        return NGX_ERROR;
    }

    if (overwrite != NGX_HTTP_MOVED_PERMANENTLY
        && overwrite != NGX_HTTP_MOVED_TEMPORARILY
        && overwrite != NGX_HTTP_SEE_OTHER
        && overwrite != NGX_HTTP_TEMPORARY_REDIRECT
        && overwrite != NGX_HTTP_PERMANENT_REDIRECT)
    {
        r->err_status = NGX_HTTP_MOVED_TEMPORARILY;
    }

    location->hash = 1;
    ngx_str_set(&location->key, "Location");
    location->value = uri;

    ngx_http_clear_location(r);

    r->headers_out.location = location;

    clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);

    if (clcf->msie_refresh && r->headers_in.msie) {
        return ngx_http_send_refresh(r);
    }

    return ngx_http_send_special_response(r, clcf, r->err_status
                                                   - NGX_HTTP_MOVED_PERMANENTLY
                                                   + NGX_HTTP_OFF_3XX);
}