三剑客

grep：查找文件中符合条件的字符串（过滤）

常用选项

使用grep命令可以查找文件内符合条件的字符串
命令语法：

grep [选项] [查找模式] [文件名]

命令中各选项的含义如下表所示：(标红为常用选项)

例：
（1）在文件xdn中搜索匹配字符'xdn'

#编辑文件
[root@localhost ~]# vim xdn
#查看文件内容
[root@localhost ~]# cat xdn
xdn
lyx
zjx
#搜索匹配字符'xdn'
[root@localhost ~]# grep 'xdn' xdn
xdn

（2）显示所有以d开头的文件中包含'test'的行数据内容

#编辑d1,d2的内容
[root@localhost ~]# vim d1
[root@localhost ~]# vim d2
#查看文件内容
[root@localhost ~]# cat d1
1
test1

[root@localhost ~]# cat d2
2
test2
#显示所有以d开头的文件中包含'test'的行数据内容
[root@localhost ~]# grep 'test' d*
d1:test1
d2:test2

（3）在/root/lyx文件中找出以b开头的行内容

[root@localhost ~]# vim /root/lyx
[root@localhost ~]# cat /root/lyx
aaaa
dd
bbbb
bdfgtg
bbbbb
bbb

[root@localhost ~]# grep ^b /root/lyx
bbbb
bdfgtg
bbbbb
bbb

（4）在/root/lyx文件中输出不是以b开头的行内容

[root@localhost ~]# grep -v ^b /root/lyx
aaaa
dd

(5) 在/root/lyx文件中输出以b结尾的行内容

[root@localhost ~]# grep b$ /root/lyx
bbbb
bbbbb
bbb

（6）查找/etc/passwd包含root的行内容

[root@localhost ~]# grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

[root@localhost ~]# grep -n root /etc/passwd  #显示行号
1:root:x:0:0:root:/root:/bin/bash
10:operator:x:11:0:operator:/root:/sbin/nologin

基本模式正则匹配

匹配字符

. : 任意一个字符
[a-Z]：表示一个字符
[^123]：匹配一个字符，这个字符除了1、2、3以外的所有字符
tab,space 等空白符[[:space:]]

匹配字数

{m,n}：匹配其面前出现的字符至少m次，至多n次
？：匹配其面前出现的内容0次或1次，等价于{0,1}
：匹配其面前出现的内容任意次，等价于{0,},所以"."表述任意字符任意次，即无论什么内容全部匹配

位置锚定

^：锚定行首
$：锚定行尾。技巧："^$"用于匹配空白行

分组及引用

(string)：将string作为一个整体方面后面引用
\1：引用第1个左括号及其对应的右括号所匹配的内容
\2：引用第2个左括号及其对应的右括号所匹配的内容
\n：引用第n个左括号及其对应的右括号所匹配的内容

案例

1. 在/etc/selinux/config下，忽略#开头和空白行

[root@localhost ~]# egrep -v '^#|^$' /etc/selinux/config 
SELINUX=enforcing
SELINUXTYPE=targeted

2. 显示/etc/passwd文件中不以/bin/bash结尾的行

egrep -v '/bin/bash$' /etc/passwd

3. 找出/etc/passwd文件中两位数或三位数

egrep ':[0-9]{2,3}:' /etc/passwd

4. 匹配test文件中'a123456XXXXXXa123456'的行

#查看test文件内容
[root@localhost ~]# cat test 
a

 #abc

spc

  klfswc
lfwe

        lfjks
lfwfweio.28     03*#&fjlwlfwfweio

  aa123456tejklsfaa123456lsl
  aa12345tejklsfaa123456lsl
  aa123457tejklsfaa123456lsl
 aajfkefjxawekl


ddddd
        1111111d

#匹配
[root@localhost ~]# egrep '(a123456).+\1' test  
  aa123456tejklsfaa123456lsl

awk数据处理引擎 （格式化）

• 特征：

1. 基于模式匹配检查输入文本
2. 逐行处理，并将结果输出到屏幕

基本格式： awk [选项] '{指令、条件}' 文件

• [选项]常见包括
  -F：指定分隔符
• {指令}包括
  (1)print：输出
  例：

[root@localhost ~]# vim lyx.txt
[root@localhost ~]# cat lyx.txt
123
r
d'grrjioe
ejfioweuoi
wejr928309wodjse
jdferu9eijofnw43po

[root@localhost ~]# awk '{print}' lyx.txt  #输出文件lyx.txt所有内容
123
r
d'grrjioe
ejfioweuoi
wejr928309wodjse
jdferu9eijofnw43po

内部变量

$n:指定分隔符的第n个字段
$0:当前读入的整行文本的内容
NF:记录当前处理行的字段个数（列数）
NR：记录当前已读入行的数据（行数）
例：

[root@localhost ~]# vim xdn.txt
[root@localhost ~]# cat xdn.txt
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

#输出第一和第三个字符，默认分隔符是空格或tab
[root@localhost ~]# awk '{print $1,$3}' xdn.txt
hello 
123 
hello du
hello xin
Fortune the
Life too
Only who
It's to
I thee

#指定o为分隔符
[root@localhost ~]# awk -Fo '{print $1,$3}' xdn.txt
hell rld!
123 
hell  du nan
hell 
F rs the b
Life is t  sh
Only th  capture the m
It's better t ught and l
I argue thee that l rtality

#$0当前读入整行的文本内容，NF当前处理行的字段个数（即列数），NR当前已读入的行的数量（行数）
[root@localhost ~]# awk '{print $0,NF,NR}' xdn.txt
hello world! 2 1
123 1 2
hello xiao du nan 4 3
hello jia xin zeng 4 4
Fortune favors the bold 4 5
Life is too short for long-term grudges 7 6
Only those who capture the moment are real 8 7
It's better to have fought and lost, than never to have fought at all 14 8
I argue thee that love is life. And life hath immortality 11 9

数组

定义：数组名[下标]=元素值
调用：数组名[下标]
遍历：for(变量 in 数组名){print 数组名[变量]}

[root@localhost ~]# awk 'BEGIN{class[0]="java";class[1]="python";print class[0],class[1]}'
java python

• [条件]包括
  （1）过滤机制
  • 语法：
    BEGIN{...}：预处理
    {...}：逐行处理
    END{...}：行后处理
    例：

#预处理不需要指定文件
[root@localhost ~]# awk 'BEGIN{a=23;print a+2}' 
25

#a初始为3，每处理一行a加1并输出a,a在最后的结果为9
[root@localhost ~]# awk 'BEGIN{a=3} {a++;print a}END{print a}' lyx.txt
4
5
6
7
8
9
9

正则表达式(//)

# 列出所有以root开头的行
[root@localhost ~]# awk -F: '/^root/{print}' /etc/passwd 
root:x:0:0:root:/root:/bin/bash

匹配或不匹配（匹配！~不匹配）

#列出第一个字段不以root开头的行
[root@localhost ~]# awk -F: '$1!~/^root/{print}' /etc/passwd
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
systemd-network:x:998:996:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:997:995:User for polkitd:/:/sbin/nologin
tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
ops:x:1000:1000:ops:/home/ops:/bin/bash
nginx:x:996:994:nginx user:/var/cache/nginx:/sbin/nologin
tomcat:x:53:53:Apache Tomcat:/usr/share/tomcat:/sbin/nologin
jiaxzeng:x:1001:1001::/home/jiaxzeng:/bin/bash
mysql:x:995:1002::/home/mysql:/bin/bash

数值比较，逻辑比较，运算符

==　　等于

=　　大于等于
<=　　小于等于
　　　大于
<　　　小于
例：

• 输出奇数行

[root@localhost ~]# awk 'NR%2==1{print}' xdn.txt
hello world!
hello xiao du nan
Fortune favors the bold
Only those who capture the moment are real
I argue thee that love is life. And life hath immortality

• 输出/etc/passwd文件中第三列数字大于80的行

[root@localhost ~]# awk -F: '$3>80' /etc/passwd           
nobody:x:99:99:Nobody:/:/sbin/nologin
avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
systemd-network:x:998:996:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:997:995:User for polkitd:/:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
ops:x:1000:1000:ops:/home/ops:/bin/bash
nginx:x:996:994:nginx user:/var/cache/nginx:/sbin/nologin
jiaxzeng:x:1001:1001::/home/jiaxzeng:/bin/bash
mysql:x:995:1002::/home/mysql:/bin/bash

• 输出/etc/passwd文件中第三列数字大于80的行，只输出第三列

[root@localhost ~]# awk -F: '$3>80{print $3}' /etc/passwd 
99
170
999
998
81
997
89
1000
996
1001
995

(5)分支结构
- 单分支：if(条件){指令}
- 双分支：if(条件){指令}else{指令2}
- 多分支：if(条件){指令}else if{指令2}...else{指令N}

例：分别统计UID大于500，小于等于500的个数

[root@localhost ~]# awk -F: 'BEGIN{i=0;j=0}{if($3>500){i++}else{j++}}END{print i,j}' /etc/passwd 
7 19

[root@localhost ~]# awk -F: 'BEGIN{print "大于500","小于等于500"}BEGIN{i=0;j=0}{if($3>500){i++}else{j++}}END{print i,j}' /etc/passwd      
大于500 小于等于500
7 19

sed：流式编辑器（修改）

• 特征：

1. 基于模式匹配过滤及修改文本
2. 逐行处理，并将结果输出到屏幕

• 基本格式：sed[选项]'编辑指令'条件
• [选项] 包括：
  • -n：将匹配结果输出（默认是全部文件+匹配到的文本，加了n就只输出匹配到的文本）
  • -i：直接修改文件内容
  • -r：扩展的正则表达式，若与其它选项一起使用，应作为首个选项
    例：

[root@localhost ~]# cat xdn.txt #查看xdn.txt文件
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality
 
[root@localhost ~]# sed -n '/^L/p' xdn.txt  #匹配以L开头的行，并将结果输出
Life is too short for long-term grudges

[root@localhost ~]# sed -i '/^L/p' xdn.txt  #匹配以L开头的行，并将结果直接修改到文件

[root@localhost ~]# sed -n '/^L/p' xdn.txt  #匹配以L开头的行，并将结果输出，此时已经有两行了 
Life is too short for long-term grudges
Life is too short for long-term grudges

打印指令（p）

[root@localhost ~]# sed -n '4p' xdn.txt #输出第4行
hello jia xin zeng

[root@localhost ~]# sed -n '4!p' xdn.txt #输出除第4行以外的行
hello world!
123
hello xiao du nan
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# sed -n '2,4p' xdn.txt #输出第2-4行
123
hello xiao du nan
hello jia xin zeng

[root@localhost ~]# sed -n '2,+3p' xdn.txt #输出第2行和后面的3行
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold

[root@localhost ~]# sed -n '$=' xdn.txt #输出总行数
10

删除指令（d）

[root@localhost ~]# sed '1,3d' xdn.txt #删除1-3行，只是在输出的结果中删除，不会影响源文件，若需要修改原文件，就要加-i
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# sed '/^h/d' xdn.txt #删除以h开头的行
123
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# sed '$d' xdn.txt #删除最后行
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
[root@localhost ~]# sed '/^$/$d' xdn.txt
sed: -e expression #1, char 5: unknown command: `$'

[root@localhost ~]# sed '/^$/d' xdn.txt  #删除空行
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

替换指令(s/old/new/n)

[root@localhost ~]# sed 's/h/H/' xdn.txt #将每行中第1个h替换成H
Hello world!
123
Hello xiao du nan
Hello jia xin zeng
Fortune favors tHe bold
Life is too sHort for long-term grudges
Life is too sHort for long-term grudges
Only tHose who capture the moment are real
It's better to Have fought and lost, than never to have fought at all
I argue tHee that love is life. And life hath immortality

[root@localhost ~]# sed 's/i/I/2' xdn.txt #将每行中第2个i替换成I
hello world!
123
hello xiao du nan
hello jia xIn zeng
Fortune favors the bold
Life Is too short for long-term grudges
Life Is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is lIfe. And life hath immortality

[root@localhost ~]# sed 's/i/I/g' xdn.txt 将所有的i替换成I
hello world!
123
hello xIao du nan
hello jIa xIn zeng
Fortune favors the bold
LIfe Is too short for long-term grudges
LIfe Is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love Is lIfe. And lIfe hath ImmortalIty

[root@localhost ~]# sed -i '1,3s/^/#/' xdn.txt #给1-3行添加注释
[root@localhost ~]# cat xdn.txt               
#hello world!
#123
#hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# sed -i 's/^#//' xdn.txt #解除注释
[root@localhost ~]# cat xdn.txt            
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

行前插入文本（i）

[root@localhost ~]# sed '2iYY' xdn.txt #在第2行前面插入文本YY
hello world!
YY
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# sed '2,4iYY' xdn.txt #在第2-4行前每行插入文本YY
hello world!
YY
123
YY
hello xiao du nan
YY
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

后插入文本(a)

[root@localhost ~]# sed '2aYY' xdn.txt #在第2行后面插入文本YY
hello world!
123
YY
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality


[root@localhost ~]# sed '2,4aYY' xdn.txt #在第2-4行后每行插入文本YY
hello world!
123
YY
hello xiao du nan
YY
hello jia xin zeng
YY
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortalit

替换当前行（c）

[root@localhost ~]# sed '2cYY' xdn.txt #将第2行替换为YY
hello world!
YY
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# sed '2cXX\nYY' xdn.txt #将第2行替换为XX'换行'YY   \n为转换符
hello world!
XX
YY
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

读取文件到当前文件(r)

[root@localhost ~]# cat lyx.txt
123
r
d'grrjioe
ejfioweuoi
wejr928309wodjse
jdferu9eijofnw43po


[root@localhost ~]# sed '2r lyx.txt' xdn.txt #在第二行下方插入lyx.txt
hello world!
123
123
r
d'grrjioe
ejfioweuoi
wejr928309wodjse
jdferu9eijofnw43po
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

保存文件(w)

[root@localhost ~]# sed '2,3w cc.txt' xdn.txt  #将第2-3行另存为文件cc.txt
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

[root@localhost ~]# cat cc.txt
123
hello xiao du nan

复制行、剪贴行

[root@localhost ~]# sed '2H;$G' xdn.txt #把第2行复制到文件结尾
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

123


[root@localhost ~]# sed '2h;$G' xdn.txt #把第2行复制到文件结尾，注意没有空行了，这种情况最常用
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality
123

[root@localhost ~]# sed '2H;$g' xdn.txt  #把第2行复制到文件结尾，注意原本最后一行被覆盖了     
hello world!
123
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all

123

[root@localhost ~]# sed '2h;2d;$G' xdn.txt #把第2行剪切到文件结尾
hello world!
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality
123

[root@localhost ~]# sed '1,2H;1,2d;$G' xdn.txt #把第1-2行剪切到文件结尾（这里不方便的是，用H会带一个空行，但如果用h，因为前一行会被后一行覆盖，所以无法剪切多行）
hello xiao du nan
hello jia xin zeng
Fortune favors the bold
Life is too short for long-term grudges
Life is too short for long-term grudges
Only those who capture the moment are real
It's better to have fought and lost, than never to have fought at all
I argue thee that love is life. And life hath immortality

hello world!
123

本栏目推荐文章
• 【Shell基础】Bash基础与Linux三剑客
• Redis缓存三剑客
• Linux文件查找、三剑客、正则表达式
• Linux三剑客——awk
• 测试用例设计方法六脉神剑——第三剑：倚天屠龙，正交试验冲锋
• Linux三剑客
• 文本三剑客——sed
• 正则表达式与文本三剑客——grep awk sed
• Linux三剑客-sed
• Shell必备三剑客

三剑客三剑客shell 三剑客16文件awk 三剑客linux grep 三剑客 文本linux awk 三剑客 文本linux grep 三剑客 文本sed 三剑客 文本 三剑客gkctf 2020 三剑客linux sed 三剑客 缓存redis