From c2c64016e4f4ff2cc04beb2cbc6983dd3063ca0c Mon Sep 17 00:00:00 2001 From: Bharath Veeranna Date: Wed, 5 Nov 2025 09:24:53 -0800 Subject: [PATCH 1/3] Create Vxlan_kernel_routes.md --- doc/vxlan/Vxlan_kernel_routes.md | 270 +++++++++++++++++++++++ images/vxlan_hld/vxlan_kernel_routes.png | Bin 0 -> 19821 bytes 2 files changed, 270 insertions(+) create mode 100644 doc/vxlan/Vxlan_kernel_routes.md create mode 100644 images/vxlan_hld/vxlan_kernel_routes.png diff --git a/doc/vxlan/Vxlan_kernel_routes.md b/doc/vxlan/Vxlan_kernel_routes.md new file mode 100644 index 00000000000..d985dbf0884 --- /dev/null +++ b/doc/vxlan/Vxlan_kernel_routes.md @@ -0,0 +1,270 @@ +# Vxlan configs for CPU traffic +# High Level Design Document +### Rev 1.3 + +# Table of Contents + * [List of Tables](#list-of-tables) + + * [Revision](#revision) + + * [Scope](#scope) + + * [Definitions/Abbreviation](#definitionsabbreviation) + + * [Overview](#overview) + + * [Requirements Overview](#5-requirements-overview) + + * [Architecture design](#6-architecture-design) + + * [Cofiguration and management](#7-configuration-and-management) + +# 1 Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | | Bharath Veeranna | Initial version | + + +# 2 Scope +This document is an extension to the VxLAN feature implementation defined in [VxLAN HLD](https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md). This documents specifically deals with kernel routes and interfaces that are required by the CPU to communicate to a VxLAN endpoint. This is for a specific use case where CPU generated packets (such as BGP, ping etc) shoud be encapped/decapped with VxLAN. Transit traffic (which are not destined to CPU) are not in the scope of this document. NPU config required for transit traffic are discussed in [VxLAN HLD](https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md). + +# 3 Definitions/Abbreviation +###### Table 1: Abbreviations +| | | +|--------------------------|--------------------------------| +| BGP | Border Gateway Protocol | +| VNI | Vxlan Network Identifier | +| VTEP | Vxlan Tunnel End Point | +| VNet | Virtual Network | + +# 4 Overview +This document provides information about kernel routes required for SONiC to encap/decap VxLAN traffic originated/destined to CPU. For scenarios where SONiC needs to communicate to an endpoint that is behind a VTEP, the kernel needs to be aware of the VTEP and have routes to encap/decap the packets before sending it over the wire. For example, if SONiC needs to establish BGP over VxLAN, the kernel should know the VTEP and overlay routes to send and receive the packet. If the kernel is unaware of the VTEP, it will treat it as unreachable and drop the packets in kernel. + +Currently, SONiC creates kernel routes, bridge and vxlan interfaces for a VNET. For example, consider a VNET `Vnet_1000` as defined below: + +``` +--- CONFIG_DB + |--- VNET + | |--- Vnet_1000 + | |--- VNI = 1000 + | |--- source_tunnel + | + |--- VNET_ROUTE_TUNNEL + |--- Vnet_1000|10.0.0.2/32 + |--- endpoint = 100.100.100.1 + |--- vni = 2000 + +--- Kernel + |--- Vnet_1000 + |--- Brvxlan1000 -> A bridge for Vnet that terminates Vxlan and does L2 forwarding + |--- Vxlan1000 -> vxlan interface +``` + +For the above config, SONiC creates kernel configs for a L2 bridge and a VxLAN interface. For the vxlan routes that are added using `VXLAN_ROUTE_TUNNEL`, there are no kernel configurations applied. The kernel cannot initiate communication to the vnet endpoints behind VTEP since the kernel interface and routes for these prefixes are not installed on the kernel. This document enhances the VxLAN capabilities of SONiC to have the kernel routes and vxlan P2P interface to communicate with the remote endpoints defined in `VNET_ROUTE_TUNNEL`. This can be used for traffic originated by CPU (like BGP, ping etc) and destined to a remote VTEP endpoint. + +Additionally, SONiC may need Loopback interfaces attached to the VNET which can be used as the overlay source for any communication to external VTEPs. + +# 5 Requirements Overview +## 5.1 Functional requirements +This section describes the SONiC requirements for Vxlan kernel interface and routes required for the OS to handle VxLAN encap/decap for traffic originated/destined to CPU. + - SONiC should be able to encap/decap VxLAN traffic originated/destined to CPU + - Processes on CPU could leverage these routes to communicate to VxLAN endpoints (establish BGP, ping etc) + +## 5.2 Config Manager requirements + +### Vnet Manager: +A new component called VnetMgr will be introduced that will handle kernel programming for `VNET_ROUTE_TUNNEL` endpoints. +- VnetMgr should handle vxlan interface creation and deletion for routes defined in VNET_ROUTE_TUNNEL. +- VnetMgr should install/delete kernel routes for the VTEP endpoints. + + +## 5.3 CLI requirements +- User should be able to specify if vnet tunnel routes should be installed on kernel. +- User should be able to bind the loopback interface to a VNET + +``` + - config vnet add-route + - config interface vnet bind +``` + +# 6 Architecture Design + +## 6.1 Config DB +Following new flag will be added to VNET_ROUTE_TUNNEL table to indicate if the flag has to installed on the kernel. By default the flag would be false. + +### 6.1.1 VXLAN ROUTE TUNNEL +``` +VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}} + "endpoint": {{ip_address}} + "mac_address":{{mac_address}} (OPTIONAL) + "vni": {{vni}}(OPTIONAL) + "install_on_kernel": "true" / "false" (OPTIONAL) +``` + +### 6.1.2 Loopback interfaces +``` +LOOPBACK_INTERFACE_TABLE:{{loopback_name}} + "vnet_name": {{vnet_name}} (OPTIONAL) + +LOOPBACK_INTERFACE_TABLE:{{loopback_name}}:{{ip_address}} +``` + +### 6.1.3 ConfigDB Schemas +``` +; Defines schema for VNet Route tunnel table attributes +key = VNET_ROUTE_TUNNEL_TABLE:vnet_name:prefix ; Vnet route tunnel table with prefix +; field = value +ENDPOINT = ipv4 ; Host VM IP address +MAC_ADDRESS = 12HEXDIG ; Inner dest mac in encapsulated packet (Optional) +VNI = DIGITS ; VNI value in encapsulated packet (Optional) +INSTALL_ON_KERNEL = true/false ; Indicates if this route should be installed on kernel +``` + +``` +; Defines schema for Loopback interface table +key = LOOPBACK_INTERFACE_TABLE:loopback_name:prefix ; Loopback interface with prefix +; field = value +vnet_name = string ; vnet name +``` + +Please refer to the [schema](https://github.com/sonic-net/sonic-swss/blob/master/doc/swss-schema.md) document for details on value annotations. + + +### 6.2.1 APP DB Schemas + +``` +; Defines schema for VNet Route tunnel table attributes +key = VNET_ROUTE_TUNNEL_TABLE:vnet_name:prefix ; Vnet route tunnel table with prefix +; field = value +ENDPOINT = ipv4 ; Host VM IP address +MAC_ADDRESS = 12HEXDIG ; Inner dest mac in encapsulated packet (Optional) +VNI = DIGITS ; VNI value in encapsulated packet (Optional) +INSTALL_ON_KERNEL = true/false ; Indicates if this route should be installed on kernel +``` + +## 6.3 Config Manager +A new config manager called VnetMgr will be added which will handle kernel routes programming for `VNET_ROUTE_TUNNEL`. + + ### VnetMgr +![](https://github.com/sonic-net/SONiC/blob/master/images/vxlan_hld/vxlan_kernel_routes.png) + +For the config below: + +``` +VXLAN_TUNNEL|{{tunnel_name}} + "src_ip": {{ip_address}} + "dst_ip": {{ip_address}} (OPTIONAL) + +VNET|{{vnet_name}} + "vxlan_tunnel": {{tunnel_name}} + "vni": {{vni}} + "src_mac": {{src_mac}} + +VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}} + "endpoint": {{endpoint_ip_address}} + "mac_address":{{overlay_dmac_address}} (OPTIONAL) + "vni": {{route_vni}}(OPTIONAL) + "install_on_kernel": "true" +``` + +the following linux kernel interface and routes will be added: + +``` +sudo ip link add Vxlan{{route_vni}} address {{src_mac}} type vxlan id {{route_vni}} local {{tunnel_src_ip}} remote {{endpoint_ip_address}} +sudo ip link set Vxlan_{{vnet_name}}_{{prefix}} vrf {{vnet_name}} +sudo ip link set Vxlan_{{vnet_name}}_{{prefix}} up +sudo ip route add {{prefix}} dev Vxlan_{{vnet_name}}_{{prefix}} vrf {{vnet_name}} +sudo ip neigh add {{prefix}} lladdr {{overlay_dmac_address}} dev Vxlan_{{vnet_name}}_{{prefix}} +``` + +# 7 Configuration and management + +## 7.1 YANG model +Yang model for vnet and loopback will be changed to include the new fields. In [sonic-vnet.yang](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vnet.yang), VNET_ROUTE_TUNNEL will include `install_on_kernel` flag: + +``` + container VNET_ROUTE_TUNNEL { + + description "ConfigDB VNET_ROUTE_TUNNEL table"; + + list VNET_ROUTE_TUNNEL_LIST { + key "vnet_name prefix"; + + leaf vnet_name { + description "VNET name"; + type leafref { + path "/svnet:sonic-vnet/svnet:VNET/svnet:VNET_LIST/svnet:name"; + } + } + + leaf prefix { + description "IPv4 prefix in CIDR format"; + type stypes:sonic-ip4-prefix; + } + + leaf endpoint { + description "Endpoint/nexthop tunnel IP"; + type inet:ipv4-address; + mandatory true; + } + + leaf mac_address { + description "Inner dest mac in encapsulated packet"; + type yang:mac-address; + } + + leaf vni { + description "A valid and active vni value in encapsulated packet"; + type stypes:vnid_type; + } + + leaf install_on_kernel { + description "Flag to install this route on kernel."; + type boolean; + } + } + /* end of list VNET_ROUTE_TUNNEL_LIST */ + } +``` + +The yang model for loopback interface [sonic-loopback-interface.yang](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-loopback-interface.yang) will include vnet_name field: + +``` + list LOOPBACK_INTERFACE_LIST { + key "name"; + + leaf name{ + type stypes:interface_name; + } + + leaf vrf_name { + type leafref { + path "/vrf:sonic-vrf/vrf:VRF/vrf:VRF_LIST/vrf:name"; + } + } + + leaf vnet_name { + type leafref { + path "/svnet:sonic-vnet/svnet:VNET/svnet:VNET_LIST/svnet:name"; + } + } + + leaf nat_zone { + description "NAT Zone for the loopback interface"; + type uint8 { + range "0..3" { + error-message "Invalid nat zone for the loopback interface."; + error-app-tag nat-zone-invalid; + } + } + default "0"; + } + + leaf admin_status { + type stypes:admin_status; + default up; + } + } +``` + + diff --git a/images/vxlan_hld/vxlan_kernel_routes.png b/images/vxlan_hld/vxlan_kernel_routes.png new file mode 100644 index 0000000000000000000000000000000000000000..89279dbce751f172923680f5f980c60b5280223f GIT binary patch literal 19821 zcmeHvc|cP6`nNTUP0eUCQ`u&WEmKx5ZCY-W)tF0CSedy{nVI{PBBBD7m6c7DsaU48 zh=v=vlAut{lo~FjC@LbAfdL{33bDTj8?Ao#z3*@Cy}x(Ae{|v;&w0-G`+T4E^E}^k zVVAetchl!jH!?E%&V9!=sFBeZaNwWSw6B3H8$0$>fd8h%Lfu@A%32o+fRisnx9r?v zWK@owp*i#waQ@A&JN#mejOHve{F~wqUHZ<*Xy;V-ZCmyv1c^B@_W}f{#9NV>hArI$H zO-E=Sx3quRvUDA-U-Rf#Vvab^jP{NEb?)fmCw3mBq+IwVx4S*eGnG!Pv3uCd-g|Q? zdkck{yg}KilxVHvAQR8k=8=kq>qF-QBYg}N0>`O7fBCo|K3zwyAgv$84gB9-R6YLrG~K|L}WPBcoV+>?%OTwcKDZWl0$_@EB#`XQXaqg`nr~4 z63qdua2xKc?YQZPC`0O$gE?x2q^*rZle^^e6d1i?5LY)i#)4rIl-+8dAI4uE8Y5wH z9vmOYUmaL`A_U!veIp|UQtiN@u_)0{7RA5_2~9n$Y45~UW#NgH^4EX9i%=$6l5M)k zDItF|Pn3^!LF;6Au%(|s1$zI`c#$QFtCN#+ihW6W0bR5@b+ms_yS$e$tQr$7A2g8M z{*K?0gUzIjmN(bgAl*gCf#2*G4ND5aT0ujX-;9ZuSPRf!uA@6e2PeZ*X8{y4r?(?zC@?I1GL-GQN0W_6n{us|o*h^v`!@agNGLFmD< zKw-PhTK8pTNjVyZ*NG-ivODNiBq_AB!N%t*gY3>~5_z31Q4b%YttJ(tDD`x_b|@5L zO}hGlQ7kCSdJ9bR+6+Q#mYaX5#Hl2a*2=dgr67hojrQMK9O7P$@U-h4peaH zb@UAeF+Yfq9}uPoJdoNf4Z{Q+-$4}7d|08tKnr8tMd-S9F*aIFkV7sB7mep;`UugZ zb8mt%K?b2R?@1yu2$r~f&G3P>`Am2W9+!-4N7=``poWV@ys6!sAWG|<_I*i2PX$>2 z_=a3I&B*9dsDxnM1a9lHKHPK#ipxi|wL79}ufSQE(Xz)gY;NAfvpUJcnitPP*b#Ie zgrJ>pjTa13?~fIAl@@shQlFbITGZMGn3Jb=7ocYDI>$sos1X;f7V~qs8ov9XlJdro z>E=d8Ls%&JoW?s~r45Imwhp$MWl#(oLiA1)Xv_;l*F3eSt)c(oJboEWxa_n0Ex?NT z84g_q&!-qYj9URO*?t?CW7uxtTra~90q|qn_!X#dzx((V;D^Wf6(8aL9phIPlrmAH zcR^9KcV}((Pqpza3m7N+`WH($#CeSAKJWe?FvA=hb_)M=xzqwy0K}j6HS$xpH z(AM>=vcMdCiF}F}!}THP^8xRW7Xrt1DYP zZR(lH0}Hs>FqtzWTK2>SeH>jn^3Co2cGpROL*l*JAQu*sTHBBTd4-eCtB(`W)?n=N z?o_5)`tnf<|EyWi##$XsQQwkWegQt{+Ta&!x+ z8s%ZWx{z7^V#{FIo0#UX9^W0jFP`sd3_>e;sh-p-c*crOe1TvS4g;cym|>oKe~T|seN=`LP8+nGPJ}|a`mWFm_D@m zM~Ksly7#ZsmixU2Jg4TBQ=xW;5C z+88Ba-(kOzA>apBo;loegmzA2in4a2X5Y0SSey4*Ij-=MCc2DVfOrYb&;Kzo-9gk& zwUh`!3CkIDtvCi;mwZY8W{>dTCc@?1EXu;U#JOTA21MN#<(TT`c+8CRvvJ_cqL})z zE=}LaG=SH+q!8n{ttQjRVLOPPvU1^1>)U))-BN1>W*VS7-s>d0k2x!(MgRAJn z`(_hMoP)#WwDSaUH?VDJd8I7Ed1d`Mj6{K!yn=V6qg@KLl>}udhT_>Dp9ITl@l6T> z3^)1L_;v?rL*1OHbzxj?9H^Zopy`_MnkLPB5upFC*UJ`#JWbVTVC{%{i`X0Tkq*3W zv?)X>jYki{^P#L(Wr!ka_P9WL&vq@$r1g+ZUrP16RxQ#+2N9E z79Q>X97osi_%gg+(=^6rdx1(-17O|2HCcy^D^gs|@-E9Uoz9lMu#QHBSOwaM8}5jB z>MP`qemwYn3OM1-=JhapD}i)$or_?}Ee~_`uQ!lb+X05Wyx4~2+Y*LOwGGUx$(1_l z3%!FNN#F&ueTocN%Z zMb*l;HQT2>srXGPblh=Z@A*0Pd}?JE;(7LHOJfC697J~n`>C+GbunqfTNPkTdYx`q zHHX9PSPO8sO6D>)uWb;aAhcCL$;Z6a(bw|tC_72$u{R>PL@|?}o4x`*RKm2Ho+aNB z#J-dAIDdhWD8ODYqq1yRERAYMa+ZnpVPIXkHNZ(bezD|M$cu=C7pq6jf8xv9F&-Ko zOcx@Pvtz&kG-XRno(6(U$`NHyXd=E$3eOVzAQo`gM;S-;>j6u>QCtT>J(VeQvM97o zVz$IZ)F-(;ZvDI6i4mBN7LcIPQTd5nExQ!G^F410<^)~Bhcj~!=jxWN55?|D90o%If(o% z-{=^g4)8hTD=S!5mwnYVzF6_qs~W(7*KC3*2O%9X$s$@oCEPT*S7H*U6?1h_v5H}i zlE|y9JLV!)b+p$9a#hmi0$7O4A9vJA2l#0_BON3@c_U6aXt0{mPHvL?yn6b}s2>3#h^6(CWZU(p?hl`Ws5e__DYDVy4Wkh)GH#l%Y&j^IfM{>O z#FJLvV^*`c^(8TQDN>*kFKY?BAxb`pv!c)oLulQ96v$k7^B~fjg$y4)0lf2h>m%CO zvSNN!eSZ~4?jTeAHmMj*Ke-nIs~O#Haz&nQvao~u zC|utbQuEd>Y=Pwpwaa`vgYmH&jOf&7EHB*j&K(EW+a7ngFPw94sdy`QO5YtQZnbLB z{bTrc#~EQ|?W_5)d!uZ+mTjsa)#vRsP7?tNe7FONmtG1LaO$hdG!68Qij~Hx&0&eY z)><(|?qH@J8L62xk2Q(UYj=tkmPDx^Mn5~VYqd`by?ymbtRD_4FW}#QV=?Qz@?!M| zHd&1u-u0nvc~Px5HwAxFmHhs9P#!qRD_41uoNeRHsB7?Vb|mZ)9@A;s^mWQ=n7aCP zR+^cO9at{|bCdku@a!=%`y6A_Ma_8Ug4X0G2R=l?bH+X8#mt}-*!#z3{KAt_Yhk-g zWhNbk%-yRkz8qNdPn?ycq@?iQzCDJX-u<)et1(-NEf9k}xzTb|Mmia_Zi3(x0M064 zetO(p^84?<^WVQu`20Z+(iZyZc~1|~ZY|=Q7eI)2_q$Ge8vL)H85TKKRj?4uLp|2AE`yFWEGwPj>v!?_xq?Nd{%^AP78f%P0Iv%0N0s0j_ynz-O5c& zO>+hY20njqRsGl6=||_LDo-b7BeGoxmk)nqF&F!q!S$<_e!Odc;E2CxUahK7dEAKaJtooaTot;t8>`N7q8(Z7D? zuyNzD%a<>IF6AXXk#_p&-FB1Yjf^%Q^Z@XVkH;qtPEL`RvDm@S)T3l{{^+XgLkm9& z&$2TIo&NPR_^-b{UubUL0-3@-G9iX{_gD5r>So`Mnj{ZET@MY=eb`asXFox1f9qjx@r^~d zP30FM0JX0Pm#>_eIfUyuwYJe`MGDkyhwC&sV`%fdHqPw0I9s*w0w-%4jayT$xIedcNH zTE|^qPf&VF`1i(pL!?#uMc9|yJxFN>j5O{I{via z>cn%^I|wh8gZP+XE*%UWbf&KO7Dy3~0mfR^ctYG;tt6<|D67Nol9rTU`5aQA~ z)SYQ<8EMx(qwz-CWwo0)l0DcZ?k3>YmWbs zV;_?@>*474e+*NTIM~&E*KWn`r}5ta;q07Jdj^03?`}Rj+FzgKNv;p)_>nhS0ag)3 zYrWGD*~#lELhTV|JA5QMdbL3|*alubi-Qf~AAVtNE?%5%1KZa%Clk9JME7D3`^OxB zz>o*_-B;B(AHi1+i&G=l!9qKOYu-waG`h$+_Srvtq!zG9>Sk*;J7fE@0{`j2fNg^6 z-tRahSOp^Hzd_ulNVdThOZ@h1WLd$t$~TGn(T#S#fuW5KcF2H3D<_0yTU4*ah;Zk* zZXOj#rR%9()}hEIvfn--MH0m8;G|!RtYnt<4o2b$!&TT-US}ud;@(cQ?_0T;jji{u zE;>0{f0wx-pSi5IF+U3K>AjZk2u~ZTWai3`pN&$#GR1V)$yyy{KEF?{Wbd2w;{e3! z=CN;W)veX5%!tWxteiN{!1Oc^>tDrV^3uY5d8u*A9}u5)V41w_HD3T{#Ze`} z$d)>&FCJ1mgyqE0lt*Plm2G8x@*-5L>S9MxR9Ry4Ij4dq!>V;>?h#5Oe$73xd_s?n z5+GD&sn75P=e{(O2v)(7e)~SX0x(Y6?)&MLc~IbrHR6M=wbwizSOQY+kd1ag@O4M6 z)P6xp;%a5(6y{qTMu|(J+wbr4&qMOB_fI89y7Pwbg0%Ob#$WB=8fQ2>?%9`6H`?ym zGA3d{bg3N`_+{FtWw(L^oI5nhcUfauqijF*MfM$D3P^L%fLfRGh)`oS!Jy$&Pb%^c zvwZt0p4PF~CzjCYrE{@IAh|hY!Vav2sNU}4g36BGXa{wA8(&mnuR#2`pzlJ#Qqb8L z{cc1}@NT029{$=1ZfAzrPmahCbfM@9*k1N|;sFet4JJri%C4}Uko9(dDt(dByUYu1)7*HR9x+iou`U zC0`QM*K8n|nx&-B#(|1RP=#C@SOeq02_cOU(H--4^r{~;&n2}Chj6qz)bR$ZXDN4| zFzLKTV~qHVx@gN|u;myItqx^$(S*yEv-&b$h?J%df4*G-aH_%NBAN z$!6XO*c<_y8^@NPLLl(z7HqFJ2m(aGhN)?BHvUICYt+LXMGYlMAYW%|l=@1|3-qOQ zIck0dY%3@w^4h${bJ4NaSVy@c)QjK;$LzXoCYN^!XmgEcpxVeTOn>+?Eu~3g@{7Yy zV%-nerBO@jmvp)q1u0IR=!{oa&2XpT877fb(Si>{8Fm>JcU*}wv4kTzIP1Uzs=3vr zz49xAw_?o2qLRb_RbB_Jfr!GS4zVbt+Pc<`=FN6-ImN@UcCxNalc-m*=(_IM=bc*Y zRJq$HtjQ2VaaE?OUKq*Y)=p!(@-Bk2jC+*x#n6$)Q*qoH85+-jh-^gp<;P&C>gQz0 z)|%_`%B0QxitK_xD^8iZ&$=R?wD*l9Fbuvq2hh8R*X*ee zZ0E)n<7YV;l%2z#H1Qq0xu;4V)@av(wq6D2z~dChJ6&vg<7vq+Vz0bWGj-jE6N4xu z>~^dt4V45^$;lF(HJh|u^fqZ>Z#%pz_f$zemQe$+gBNLxp7FnMzU7Yv3T-kuj-Rtfr*7jUf5V%7Y0)z@rXk!u?%Za zDhkXG48!zF`pRLGYXDItu!^vEep`lFxEiykgc=8qpL6O?U)w+h4B6$R>zEjWg;Dp- z2{U_;u2Yfp6eX3erYbFeS~;OzkI)3{nyDibE^(TX-A7mR@oA%ju@i4Ut^F8a-<$M8 z>c1Yz{pA>y_2|)-*tod$;IFn#thPlrX*>fs?=_*0e<%1f%9)H*hgh#(9g0m(cKVFY zUk``>`phQNzfalQaL#kUp6WeoE)_aQH7X@&CF&|9>zu(j-=K z-eY3@@59EEt(hO=iI1ZCcX-`RnuIRShx>1Oe-)dYk5mZ?A(y8V-ZzvpX>PkSM=8F- zX>P<6LsD`sszqw3g$2;j^|eWg;iXwX+k|S-_Ps!!)w^%nCflU#0x4$3jY5E#3xJ%{<2#Gsvi-#l1u#aW}_kKUq9V;{A2R@Ipx2=^RqouP6~jm+I@Wajcm<hRrI$!%|Pm#}T!p>;N$>0@Q5bq~c)zGcNI6yr%>3u-GN?0uO?3bxTBz!;@0)q&Q zzPmtO?cwz2zyv3!KF#pp$lAc%J~{@2ZnYIH9Ewl-quclykr|gJux5C1<>Xh7QjA;9 z&M>5UyQc=7J|b`(m%+IUoJ{f7Q)SvQ2eC(G6 z%SvU&;P;Q*%a7slBrYuqZU4p)0*fFlk#6+?MQ|#L<=&piW(n9q>E!Dq7TUL1ogK%u z@SIb>fP75NsKMn&1=J;!E8B6jxD#OYuzx>Ru6^sF&a;WZ$ zt!6juRmAkj5I4s74R^FR+;P0M>>y114ffLD`h6GN$CvI_q~IYP_3nC>oa!hzX=ipb z+M==0w?2jGyHsV8!8WFe#r%|!fTDd-F`bTZWD-Z6K4U9F8-@}DU&gDLh2GxFQ3^{@ zG^SIYo(y6Sj)ma{|vue>@d6d*4mA3DK+XtZ%y?b|mlZ#6p{U((0}Z5wjHLT*+o!R5F$&m^+0G4h6Wa_@sKoY`r5R8O;KGOU6FV! zH)t7dM4s3&kl~Ve9fFX!wJQRSpcKXRur&GcPObl#rZko-6OAQt)&0R_K|gsIJr)(( z8I`^UR{RHG$Ii)san@CA-sUSfO5b`bM z4901zVwAY6`S5%Ol}9Y5YAx8JRSS#rwL3tv2wdCfdjw8S8+A7Gt;8P6hVK@`|{8-rO;!Xd$_M1*>44f$So z6`XOT*W0>H+)!-A#cEo2tB=2Z#)>kNR<%SQ5)2O%smVg6e^q}pk-_p5-3TZ8!(kU7 z6Bfp{ChmDZ+SkHLX;h8cjDM5zI(o|v;|dPE$}~~-2x;ZKiRjZ;<#$B8g zMKVlEQJMT9vy;V3(eb7#DquyzlTc^xLMR}ld2#=DExkaL2voSqZ|xlc5Mx|j$dUD8 zHUU*@KqM)JKsOvb=##~N`2w0gW}aMmo%XomyA95mpkm%6o8I01f=5H37p{n5WfRJ> z2HPU4lm9eHlYl+b68D?Dp=aUJ818;TS;43G2#lr}Wg7rV12+2p`wu_5`y`rH$$C<~ zyjQ~%L`68W*H&v!m0QBkM?+JH#@uMRzVmf6?n7G{_a=HNe4{-*+o zza6>39m{hIW9VABrdTZf5SLU*W}fZ^V$I)a($Qj=HhjD*adq&!KmSUaITiip9s;yfAzi;rvQI2O$ zDgrR-eKH*wg%n5!^g;svx(%jCQBmil2&H!})N6B<2Tob`9?jD|w$Zr89QUtrQogyi zyx})?P+p?YlHPowD4(T$t|Ktl%wvCRK`>b~)ghM#wBx&>oh>hO@wQX=9yR`Jj^3g0aF)zZYI;4n36cO7=p`m zB$(6y5k`LCnVC!v!^O1_^|M@ng|`6~T5p{OM1NbHSl+WtB{9rLcx%yz zOa=OzMTWx*MO4y>d~U>p>r9O(cuHsn>+x8@&g)}lQJO)Pw$Er zC^;NoiOQmMxRm+i14|y>>a*ekAaWa0cl|ZkJa;i5t-E9MT!D^3*P=~lPBcA;6SOZ` zwrbviU0IoFdt4}@ofM#-dHXW62l7O9SBJHo(vAZqzjgV}a+`Q8w76!|=$Zx4Z*k&a zE^TKfAd&YrL2fSv)VOnzt8$P{!So-@o!YKDe^4#em;^Y`16@@ffd>$}uNK$Z^5F8> zzsuq6cHCrH3m=F(18c>5)i=jx7w%;TjAeg^^t!fJea|2+NqMf8zyJ+(M_{~@=*L{_ zUsQoF2X;@&Jy3@l0R+o*5+6wbn4UU`qEv%)?^0U~-OAt%fN7i1Fb~t)qYb5MU4nf_ zhVFiXDtJI?S`K*v#K}3!@yQj+=1`Z)eYFPdA`DtGT>;Tb-+Jr=Y8@Sf8tYVU^6L3p z=o;lgu+D6$D)AXMEPNAk`UZmty_xB;?3E!5??)*(%1r|vlC?W;pWK$&zt=fQKJ@!w zL=ho67qAaooIP57?itp5S?crO7TCwDUU(9s*ZJt~MZ$;-M#X=)Wf%Dfp@Y+Yg=t0Y&b^=nD%Ezl~=a@iX|DcBjWBrqNE_PZYTnh}G@&kzQMY zcl7cMRu)J|cNv^<&Fn=WWux1TLOfdaW`pC|WoBvdM8=cFc$V(TLKM%Eo`$>VuD~TM z4O<{EC&th7Cm$*4jmF2%d!~rpE%2oLJ&5`JTp(_Vs3%?6nH8FcEFeF$@`u){-(u8| zOow}CpvECPYDvuRTY&-$0T&cr2BV{;m!HncN`A++hPzW|R?`MVU?zqgcLoM$Py){N;;%1v_A`kRp0JJP@`>e;a5#jlRzKT#m!(cRRD5ZFQI#V|EFk*2D?} zHIr@HzAE_u-D3J|t%zO(Y%L#v#-X?piVa}HE4W=fWJnWGC0hvxFk7^JJctmz`#)0l zSQdf9B-hO%uC`JUEK$qire}#|c5$AZqO~|nz-^?yBxX+1Fs$wlgdPdt289x;Og^lOKNZa*p{3jB#gXnLZU+MXh)vW*a$GXhkQBUa61wOVnxF8c+iS)dBf9=?z6|JA9oegWD|u zm4824A6y%m6O5zK23Ja8T&)BuJc(L0VG0+z7WcpM+j~?M&9#~f6l%C-q$@+kOrAE; zupO)CDa|v`I!-Jisl7R2i?D&aK6~E^T;_ITb@4MV>*ZqX>H_HtYe%Gc8oI!mbhqA= zcruQ=_Zi_xE|f*SyjQZ1ATspBV>REL?8u;muzRb3md(L2g)=q`rBtDCkh7m!Xux}) zq-)D*0Y9J1xE`=kD((k-biZrWki8o}9VnOho;x9!R{?QPqMx|#x38+As!SdQR0M~z zDni%?$joKLX@d>RozVjFr5Y9&^(|W@WcTke#B=R{Vgj*l`pjN0m_={uo=B<9$$4yR z@`i0IAL!n1ydN`xqxUxN@c;@Y5x$thC>(qKVlhOWOC9I;XLpnqXkIY3{oUaApH<0! zAke5RT`RweQN!#Z7^UwGg`~PHvk}bKAVAObU)B2ivi@w9dYI zJ;!~uubu-|v|7Y{4BdT%4K}ESvldNOY5r$KIiH!Ie_*bk<`w@7So>f9x=#vbpOm~l zsUw`9c2xf!pTp3r!4W~)R(dXe-FKv9?a^3Xiyn=>=t9Cs^v7CtO*R`Ox@U-M#l=B66g z3E6-n-{JLi{$ssLgf-t*c;bXOhtvv*;ui4Sh)H|$*Vc-mQ#~xJNR2*>n*Ie$w84AF>QCUVI`E+|yi zHfaKMFG0h2;TjQJUn57&JE4apaj9+F)4S@65_oF;7)a9+%x*eYw{Hc0BHa*{552S# zQN0}wwtxyn*2+W_3+VXTS6XQW8Y*P4C9jtjcXcB0=dvh!)f)*w7zdNZ1G_pJ?nk|d z-IUK96Gw27QS23Pk&E8%0D{c4qUd4DVb+{}tOXX#o0P)9Ihj{VBt{#}?^`FPE^ace z60>jeb15Oc#_BMR7;tpJLWdnA0?BdOqPFeZ?~q8n;Ne^6@VQ55XXq+F$n1FaGa)ZRO@I>|aUgOaS(tZA#G4eQ9&63Lap z5RRJmwsr>)Thi2A)NnhTh0<7Fo8;a#)6Hzq_YC8^X(lJt(H$<&y$QB>eE$>=`pA9B z!A*hv&L`W*o{T#w`eaTWOc~ezv;scHDI8X%M0BACCPvXuIW7t5u7%;4%+P7JhBRvtd>b`2c zdM|1Avq_8npBkCJQ-nY0M3?=hCnClD3uHcp#F>F2TeHW}5 zlG|U?RcYnJ24zqEJE`pd<3?U>4Uf^M(sMD@fga{glg+BOd8kYZNe70-1ik0#a;}8{ z?XjQ!H4sM$;J0cRtCpuhjj2oiY|yZn+g4_1QUeNJNfW|-dgM}|5OCt&vJ3Y5p5PE` z8=z`&c|!OOIt6D^_VCN<#=xUOzbudidO|a@n%qfw-OR}0C zKHRS#?!rkl4U!hcw~Z4h-vWofMXm)d_p*c$QVzqKYbWPu=eD&;>}NfdJ#`B>9+d+QyM=_+5R^M zREE}n2v|GX6cP18dJU|+W~c`z+AtVC5a&)qA3JOMQe(_mYD))9m&w;+RMN-1<@y$q zof%Cks>ktoM2wcv@&ZA(GhpL@0A1nlA}+ZJ>{M9~fJT87pf|l5s3RpxLXZG5PFn|C za>e<51Ll}PDBF2LjB-`Wm|hucpya3@Ns?`t>hqzs=LxWJ1oYLZi^ zr`phhu3!7<=8o}7%t7|#mJHzI0;ep2kGFh0&YCQ}POWL0*r?%F-X8Isbnx7N!Js~$ zOxI*jN947M{T7!Gk2eZEteM=B@wcB~S>c>2Eq%Pxvl literal 0 HcmV?d00001 From 2a50d75a694da1bfe32a7ec67c0c42dd0caee688 Mon Sep 17 00:00:00 2001 From: Bharath Veeranna Date: Mon, 10 Nov 2025 16:51:11 -0800 Subject: [PATCH 2/3] review meeting changes --- doc/vxlan/Vxlan_kernel_routes.md | 302 +++++++++++++++++------ images/vxlan_hld/vxlan_kernel_routes.png | Bin 19821 -> 25823 bytes 2 files changed, 229 insertions(+), 73 deletions(-) diff --git a/doc/vxlan/Vxlan_kernel_routes.md b/doc/vxlan/Vxlan_kernel_routes.md index d985dbf0884..b47e02f8f9b 100644 --- a/doc/vxlan/Vxlan_kernel_routes.md +++ b/doc/vxlan/Vxlan_kernel_routes.md @@ -5,19 +5,27 @@ # Table of Contents * [List of Tables](#list-of-tables) - * [Revision](#revision) + * [Revision](#1-revision) - * [Scope](#scope) + * [Scope](#2-scope) - * [Definitions/Abbreviation](#definitionsabbreviation) + * [Definitions/Abbreviation](#3-definitionsabbreviation) - * [Overview](#overview) + * [Overview](#4-overview) + + * [Usecase](#5-usecase) - * [Requirements Overview](#5-requirements-overview) + * [Requirements](#6-requirements-overview) + + * [Architecture design](#7-architecture-design) + + * [Limitations](#8-limitations) - * [Architecture design](#6-architecture-design) + * [Cofiguration and management](#9-configuration-and-management) - * [Cofiguration and management](#7-configuration-and-management) + * [Test plan](#10-test-plan) + + * [Example configuration and outputs](#11-example-configuration-and-outputs) # 1 Revision | Rev | Date | Author | Change Description | @@ -33,6 +41,7 @@ This document is an extension to the VxLAN feature implementation defined in [Vx | | | |--------------------------|--------------------------------| | BGP | Border Gateway Protocol | +| P2P | Point to Point | | VNI | Vxlan Network Identifier | | VTEP | Vxlan Tunnel End Point | | VNet | Virtual Network | @@ -60,39 +69,70 @@ Currently, SONiC creates kernel routes, bridge and vxlan interfaces for a VNET. |--- Vxlan1000 -> vxlan interface ``` -For the above config, SONiC creates kernel configs for a L2 bridge and a VxLAN interface. For the vxlan routes that are added using `VXLAN_ROUTE_TUNNEL`, there are no kernel configurations applied. The kernel cannot initiate communication to the vnet endpoints behind VTEP since the kernel interface and routes for these prefixes are not installed on the kernel. This document enhances the VxLAN capabilities of SONiC to have the kernel routes and vxlan P2P interface to communicate with the remote endpoints defined in `VNET_ROUTE_TUNNEL`. This can be used for traffic originated by CPU (like BGP, ping etc) and destined to a remote VTEP endpoint. +For the above config, SONiC creates kernel configs for a L2 bridge and a VxLAN interface. For the vxlan routes that are added using `VXLAN_ROUTE_TUNNEL`, there are no kernel configurations applied. The kernel cannot initiate communication to the vnet endpoints behind VTEP since the kernel interface and routes for these prefixes are not installed on the kernel. This document enhances the VxLAN capabilities of SONiC to have the kernel routes and vxlan P2P interface to communicate with the remote endpoints defined in `VNET_ROUTE_TUNNEL`. This can be used for traffic originated by CPU (like BGP, ping etc) and destined to a remote VTEP endpoint. + +# 5 Usecase -Additionally, SONiC may need Loopback interfaces attached to the VNET which can be used as the overlay source for any communication to external VTEPs. +Consider a sample vnet configuration as below: +``` + "VNET": { + "Vnet_test": { + "vni": "1000", // VNET's VNI is 1000 + "vxlan_tunnel": "tunnel_v4" + } + }, + "VNET_ROUTE_TUNNEL": { + "Vnet_test|20.0.0.2/32": { + "endpoint": "200.200.200.2", + "vni": "1000" // Route uses same VNI as VNET + }, + "Vnet_test|20.0.0.3/32": { + "endpoint": "200.200.200.3", + "vni": "2000" // Route has VNI 2000 + } + } +``` -# 5 Requirements Overview -## 5.1 Functional requirements +In the above config, there is a VNET with name `Vnet_test` having a VNI `1000`. When this VNET is created, SONiC creates a linux bridge and vxlan interface on the kernel to encap and decap vxlan packets with VNI 1000. The above config also has two routes in the same VNET: one route to 20.0.0.2 behind a VTEP 200.200.200.2 having VNI 1000 and another route to 20.0.0.3 behind a VTEP having VNI 2000. + +Consider a usecase where SONiC has to establish BGP to both the devices: 20.0.0.2 and 20.0.0.3. CPU can initiate traffic to endpoints in this VNET which have the same VNI as the VNET since the kernel routes and interfaces are configured. For example, SONiC can send/receive traffic to 20.0.0.2 VM which is behind VTEP 200.200.200.2 using VNI 1000. So SONiC can establish BGP session with 20.0.0.2 device on the VNET. + +However, the VM having IP 20.0.0.3 is behind a VTEP 200.200.200.3 having VNI 2000. SONiC does not have any kernel routes and interfaces configured for VNI 2000. Any traffic destined to 20.0.0.3 will be dropped in the kernel since there are no routes or interfaces configured for VxLAN 2000. + +Moreover, if the CPU port is set to Egress mode in the NPU, the packets sent from the CPU are directed to the egress pipeline. Hence, the kernel has to form the packets with appropriate VxLAN headers before placing the packet in the egress pipeline. To encap the packets with VxLAN headers, the kernel should have the VxLAN interface and routes configured. + +# 6 Requirements Overview +## 6.1 Functional requirements This section describes the SONiC requirements for Vxlan kernel interface and routes required for the OS to handle VxLAN encap/decap for traffic originated/destined to CPU. - SONiC should be able to encap/decap VxLAN traffic originated/destined to CPU - Processes on CPU could leverage these routes to communicate to VxLAN endpoints (establish BGP, ping etc) -## 5.2 Config Manager requirements +## 6.2 Config Manager requirements ### Vnet Manager: A new component called VnetMgr will be introduced that will handle kernel programming for `VNET_ROUTE_TUNNEL` endpoints. - VnetMgr should handle vxlan interface creation and deletion for routes defined in VNET_ROUTE_TUNNEL. - VnetMgr should install/delete kernel routes for the VTEP endpoints. +- VnetMgr should subscribe to CONFIG_DB changes to VNET_ROUTE_TUNNEL and update the same in APP_DB -## 5.3 CLI requirements +## 6.3 CLI requirements - User should be able to specify if vnet tunnel routes should be installed on kernel. -- User should be able to bind the loopback interface to a VNET ``` - - config vnet add-route - - config interface vnet bind + config vnet add-route ``` -# 6 Architecture Design +## 6.4 Scale requirement -## 6.1 Config DB -Following new flag will be added to VNET_ROUTE_TUNNEL table to indicate if the flag has to installed on the kernel. By default the flag would be false. +SONiC will support a maximum of 2000 kernel configs for `VNET_ROUTE_TUNNEL`. Kernel config includes the vxlan P2P interface and the kernel routes for the prefix defined in the `VNET_ROUTE_TUNNEL`. -### 6.1.1 VXLAN ROUTE TUNNEL +# 7 Architecture Design + +## 7.1 Config DB +Following new flag will be added to VNET_ROUTE_TUNNEL table to indicate if the flag has to installed on the kernel. By default the flag will be false. + +### 7.1.1 VXLAN ROUTE TUNNEL ``` VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}} "endpoint": {{ip_address}} @@ -101,15 +141,7 @@ VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}} "install_on_kernel": "true" / "false" (OPTIONAL) ``` -### 6.1.2 Loopback interfaces -``` -LOOPBACK_INTERFACE_TABLE:{{loopback_name}} - "vnet_name": {{vnet_name}} (OPTIONAL) - -LOOPBACK_INTERFACE_TABLE:{{loopback_name}}:{{ip_address}} -``` - -### 6.1.3 ConfigDB Schemas +### 7.1.3 ConfigDB Schemas ``` ; Defines schema for VNet Route tunnel table attributes key = VNET_ROUTE_TUNNEL_TABLE:vnet_name:prefix ; Vnet route tunnel table with prefix @@ -120,17 +152,11 @@ VNI = DIGITS ; VNI valu INSTALL_ON_KERNEL = true/false ; Indicates if this route should be installed on kernel ``` -``` -; Defines schema for Loopback interface table -key = LOOPBACK_INTERFACE_TABLE:loopback_name:prefix ; Loopback interface with prefix -; field = value -vnet_name = string ; vnet name -``` Please refer to the [schema](https://github.com/sonic-net/sonic-swss/blob/master/doc/swss-schema.md) document for details on value annotations. -### 6.2.1 APP DB Schemas +### 7.2.1 APP DB Schemas ``` ; Defines schema for VNet Route tunnel table attributes @@ -142,10 +168,18 @@ VNI = DIGITS ; VNI valu INSTALL_ON_KERNEL = true/false ; Indicates if this route should be installed on kernel ``` -## 6.3 Config Manager +## 7.3 Config Manager A new config manager called VnetMgr will be added which will handle kernel routes programming for `VNET_ROUTE_TUNNEL`. ### VnetMgr +VnetMgr is a new config manager introduced to handle the config changes for `VNET_ROUTE_TUNNEL`. VnetMgr will do the following: + +- Subscribe for config changes to `VNET_ROUTE_TUNNEL` +- Handle kernel interface and route (create and delete) if the routes have `install_on_kernel` flag is set. +- Publish the routes to APP_DB + +The diagram below shows the flow for the route creation: + ![](https://github.com/sonic-net/SONiC/blob/master/images/vxlan_hld/vxlan_kernel_routes.png) For the config below: @@ -167,20 +201,29 @@ VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}} "install_on_kernel": "true" ``` -the following linux kernel interface and routes will be added: +the following linux kernel interface and routes will be added by the VnetMgr: ``` sudo ip link add Vxlan{{route_vni}} address {{src_mac}} type vxlan id {{route_vni}} local {{tunnel_src_ip}} remote {{endpoint_ip_address}} sudo ip link set Vxlan_{{vnet_name}}_{{prefix}} vrf {{vnet_name}} sudo ip link set Vxlan_{{vnet_name}}_{{prefix}} up sudo ip route add {{prefix}} dev Vxlan_{{vnet_name}}_{{prefix}} vrf {{vnet_name}} + +(OPTIONAL: only if the prefix is /32 IPv4 or /128 IPv6, the MAC entry will be added) sudo ip neigh add {{prefix}} lladdr {{overlay_dmac_address}} dev Vxlan_{{vnet_name}}_{{prefix}} ``` - -# 7 Configuration and management -## 7.1 YANG model -Yang model for vnet and loopback will be changed to include the new fields. In [sonic-vnet.yang](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vnet.yang), VNET_ROUTE_TUNNEL will include `install_on_kernel` flag: + +# 8 Limitations +- Linux kernel allows only one vxlan interface per VNI. There can be at most one `VNET_ROUTE_TUNNEL` with a given VNI and `install_on-kernel: true`. In other words, two routes having same VNI cannot have `install_on_kernel` flag set to true. +- Kernel interface and routes will be created for `VNET_ROUTE_TUNNEL` only if the VNI specified in the route is differnet from the VNET's VNI. This is because, when VNET is created, there is a default vxlan interface created for the VNET using the VNI specified in the VNET. Hence when the the tunnel routes are created using the same VNI as the VNET, there is no need to create another interface with the same VNI (kernel will not accept new interface since there is already an interface with the same VNI). +- If `VNET_ROUTE_TUNNEL` has overlay dmac specified, a static mac entry will be added in the kernel only if it is a /32 IPv4 or /128 IPv6 route. + + +# 9 Configuration and management + +## 9.1 YANG model +Yang model for vnet will be changed to include the new fields. In [sonic-vnet.yang](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vnet.yang), VNET_ROUTE_TUNNEL will include `install_on_kernel` flag: ``` container VNET_ROUTE_TUNNEL { @@ -227,44 +270,157 @@ Yang model for vnet and loopback will be changed to include the new fields. In [ } ``` -The yang model for loopback interface [sonic-loopback-interface.yang](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-loopback-interface.yang) will include vnet_name field: +# 10 Test Plan + +Pre-requisite: + +Create VNET and Vxlan tunnel as an below: + +``` +{  + "VXLAN_TUNNEL": { + "tunnel_v4": { + "src_ip": "10.1.0.32" + } + }, + + "VNET": { + "Vnet_3000": { + "vxlan_tunnel": "tunnel_v4", + "vni": "3000", + "scope": "default" + } +    } +} +``` +Similarly for IPv6 tunnels + +``` +{  + "VXLAN_TUNNEL": { + "tunnel_v6": { + "src_ip": "fc00:1::32" + } + }, + "VNET": { + "Vnet_3001": { + "vxlan_tunnel": "tunnel_v6", + "vni": "3001", + "scope": "default" + } +    } +} ``` - list LOOPBACK_INTERFACE_LIST { - key "name"; - leaf name{ - type stypes:interface_name; - } +The below testcases will be executed for both IPv4 and IPv6 routes. - leaf vrf_name { - type leafref { - path "/vrf:sonic-vrf/vrf:VRF/vrf:VRF_LIST/vrf:name"; - } - } +| Step | Goal | Expected results | +|-|-|-| +| Create a tunnel route with a /32 IPv4 or /128 IPv6 with install_on_kernel to true, specify overlay dmac in the route| Kernel route creation with static ARP/ND entry| Kernel vxlan interface and routes should be created with nexthop as the endpoint defined in the route. Static ARP/ND entry should be added for the route with the dmac mentioned in the route.| +| Create a tunnel route which is not /32 IPv4 or /128 IPv6 address. Set install_on_kernel to true with overlay dmac specified in the route | Kernel route creation without static ARP/ND entry | Kernel routes will be created if the install_on_kernel is true. But the ARP/ND entry will not created since the route is not for /32 IPv4 or /128 IPv6 address| +| Create a tunnel route without the install_on_kernel flag. After the route is created, update the route with install_on_kernel = true | Route update should install the kernel routes| When the route is created initially, the kernel should not have the vxlan interface and the routes. After the route is updated with the flag, verify if the kernel routes are created| +| Create a tunnel route with install_on_kernel = true. Update the route to set install_on_kernel = false | Route update should delete the kernel routes and interfaces when the install_on_kernel is removed from the route| When the route is created initially, kernel should have a vxlan interface and routes. After the route is updated, the kernel interface and route should be removed | +| Create a tunnel route with same VNI as the VNET and have install_on_kernel = true | Tunnel routes with same VNI as the VNET should not have additional kernel routes | If the tunnel route is having the same VNI as the VNET, then there is no additional kernel configs required. This is because the vxlan interface for the VNET's VNI is created during VNET creation| +| Create multiple routes using differnet VNIs on the same VNET and all of the routes are having install_on_kernel = true. All routes will have different VNIs and also different from VNET's VNI | Kernel should support multiple vxlan interfaces in the same VNET | Verify that each route get configured on the kernel with a unique vxlan interface and all the routes are programmed on the kernel | +| Create multiple routes with same VNI (but different from VNET's VNI). Set install_on_kernel = true on all the routes | Kernel should have only one interface per VNI | Verify that the first route that is created will have kernel routes. Subsequent routes created will not have kernel interface or routes added | +| Delete tunnel routes that were created with install_on_kernel = true | Kernel config should be cleaned up after route delete | Verify that the kernel routes are deleted when the tunnel route config are removed | - leaf vnet_name { - type leafref { - path "/svnet:sonic-vnet/svnet:VNET/svnet:VNET_LIST/svnet:name"; - } - } - leaf nat_zone { - description "NAT Zone for the loopback interface"; - type uint8 { - range "0..3" { - error-message "Invalid nat zone for the loopback interface."; - error-app-tag nat-zone-invalid; - } - } - default "0"; - } +# 11 Example configuration and outputs + +Consider a sample config for a VNET `Vnet500` having VNI 5000. +``` +{ + "LOOPBACK_INTERFACE": { + "Loopback20": {}, + "Loopback20|10.2.146.116/32": {} + }, + "VXLAN_TUNNEL": { + "Vxlan0": { + "src_ip": "10.2.146.116" + } + }, + "VNET": { + "Vnet5000": { + "vni": "5000", + "vxlan_tunnel": "Vxlan0", + "src_mac": "12:34:56:78:9a:bc" + } + }, + "VNET_ROUTE_TUNNEL": { + "Vnet5000|100.100.100.2/32": { + "endpoint": "10.2.146.117", + "mac_address": "00:12:34:56:78:9a", + "vni": "4000", + "install_on_kernel": "true" + } + }, + "VLAN": { + "Vlan100": { + "vlanid": "100" + } + }, + "VLAN_INTERFACE": { + "Vlan100": { + "vnet_name": "Vnet5000" + }, + "Vlan100|100.100.100.1/24": {} + } +} +``` + +For the above config, since the tunnel route for prefix `100.100.100.2/32` has `install_on_kernel` set to true, the following kernel routes will be installed: - leaf admin_status { - type stypes:admin_status; - default up; - } - } +``` +sudo ip link add Vxlan4000 address 12:34:56:78:9a:bc type vxlan id 4000 local 10.2.146.116 remote 10.2.146.117 dstport 4789 +sudo ip link set Vxlan4000 vrf Vnet5000 +sudo ip link set Vxlan4000 up +sudo ip route add 100.100.100.2/32 dev Vxlan4000 vrf Vnet5000 +sudo ip neigh add 100.100.100.2/32 lladdr 00:12:34:56:78:9a dev Vxlan4000 ``` +Output of kernel configs: +``` +admin@sonic:~$ show vnet routes all +vnet name prefix nexthop interface +----------- ---------------- --------- ----------- +Vnet5000 100.100.100.0/24 0.0.0.0 Vlan100 +Vnet5000 100.100.100.2 0.0.0.0 Vxlan4000 + +vnet name prefix endpoint mac address vni +----------- ---------------- ------------ ----------------- ----- +Vnet5000 100.100.100.2/32 10.2.146.117 00:12:34:56:78:9a 5000 +admin@sonic:~$ + +admin@sonic:~$ sudo ip link show Vxlan4000 +214: Vxlan4000: mtu 1500 qdisc noqueue master Vnet5000 state UNKNOWN mode DEFAULT group default qlen 1000 + link/ether 12:34:56:78:9a:bc brd ff:ff:ff:ff:ff:ff +admin@sonic:~$ + +admin@sonic:~$ sudo ifconfig Vxlan4000 +Vxlan4000: flags=4163 mtu 1500 + inet6 fe80::1034:56ff:fe78:9abc prefixlen 64 scopeid 0x20 + ether 12:34:56:78:9a:bc txqueuelen 1000 (Ethernet) + RX packets 10616 bytes 667730 (652.0 KiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 9387 bytes 603940 (589.7 KiB) + TX errors 591 dropped 0 overruns 0 carrier 591 collisions 0 + +admin@sonic:~$ + +admin@sonic:~$ sudo arp -a +? (100.100.100.2) at 00:12:34:56:78:9a [ether] PERM on Vxlan4000 +admin@sonic:~$ + +admin@sonic:~$ sudo bridge fdb show | grep -i vxlan +00:00:00:00:00:00 dev Vxlan4000 dst 10.2.146.117 self permanent +00:12:34:56:78:9a dev Vxlan4000 dst 10.2.146.117 self +admin@sonic:~$ + +admin@sonic:~$ ip route show vrf Vnet5000 +100.100.100.0/24 dev Vlan100 proto kernel scope link src 100.100.100.1 +100.100.100.2 dev Vxlan4000 scope link +admin@sonic:~$ +``` \ No newline at end of file diff --git a/images/vxlan_hld/vxlan_kernel_routes.png b/images/vxlan_hld/vxlan_kernel_routes.png index 89279dbce751f172923680f5f980c60b5280223f..61610391801d49dece73c6ea6b922113bdd09dab 100644 GIT binary patch literal 25823 zcmeHwc|6o>-~V)4q%1{?vUW;Rwp5HI6Hcd)rBrr`BxM`>J}n|`Dxxw>%90e>vrcJH zlu^o(#$+sGY=glt>+kvw6?N|CzJK>~p69o`oL@`1tD$U^@X7#!L(0)w6KHTXmKaG?DV!Ss@Z)p_Y-=dPVHb66y}aff1C)Ec$O zi@H8{U6*(&i>-OJ^h9Htpx|Wd&X7mvbnQo!LK=@NFSOpTqOkl->MoyClIxp}T{t7G zc439!Q}gV$#nMYQm0#1A5qQHExV6{KHYLwgb)08ANC)98Eb?_u$vG=I#XX4#!}v9 zphlHBd+9mc39Q=P1Dj*toKS`S%Vbc9>@rjfdn^HKTd169tc2_cA}0-hNHZcZsaQ|d zq_$PzwN=#52*y6{>zR9ax2E@JV7YWI$Mxop>Z7tIr!IX}xJb+$F%kehQ(Q`ZVa{A5 zE0q5L>hOJ8Ly602|3z2ml{xkm{n>V9&32`}nIWU3>VEs)J<$tcusudXhe&8N+E?AP zV^GQVxeA={9QiP0daS3vEvhgRXH_hT2^}+_tD|V@SuV(tJC!YK z!eEV*BUrBxp%p9@HVwtWJD|ExHx)$&q3C7AFm@??cj(l}cu-<<*nZ0dY3@{Q4u<iXUavyR7CCs(6(E)qR zPF^H_bz1!ui;jpaGsoy$k9BNjq)zUD_2WSjn1{y8 zZSnA28Gbn?TV6+fU$m%aj!jMJ#Bw!7L`Si!mSgmGXB(zPPHK;7FzQZZ^Dav|%TKnU zmF2W9e|S`;Hdxq7VNFKPbOn)F@o6_`&#V}tbn#jb>?)Ovn%MoT86 zeWlwyR4ZpBNpqCrbrFwcSaAEV^^mQphZVf4Zo`jDg2&!p%%_YzCcX@^HP&gk6tS7B zbX>~PnX3%k@yS(?dfdky@8w;L1jNDL#{UM{p>X}^R|@i8;OeWfp~T63=Ya0PJk3ce z+juvNrQ?pP@$Pw+ED~j9_7!~l;-t$_66-4;qK`vT=2V?npTzkp%-O@cb7Mm)E{IMD zFg2|+PS1?xJwi<@R)j3gd4&oLKExSp+Y#&fVG(G?&406OgW3+rkRL<;D>l*a-?paz zbLT(vH~&Xw2)%Hs33$2T3&J`ot5w-uC{|1v@Nk@#{tUkMoAkWRK`zo`_RX89gZdry z_qvnrZ9xisO$z9sDA|>Mtl)B(>A$68>2yv9<%MwBjlcfAi~yqp;~jAhv2*I#@*w3a z<@v%tUd=^i8y1L-*E=>s4_d=}+_%?^&4%cODt!sn_whR26$)bwb6!o1_WJe z00SEC@4P{Od;R!Fv=3|vJ5Ox9Evh_WZ^zu%%xDoZOn~9My>IT&X0HGL8Q|&lKWQymJYKZkRA;z@o0m&0yyCW1(V3B_9e+)e~5s!2qy16%)< zIZkOB9_y*kDc)wn?Z&WTn}(Q)*y+SiJBS4oKgiu4#eYr)5-}J{*9sU$bA%9~Q(k`f z_9{y+TJJkvPKEp!{~e1M3xRFm;)JWV5TbdBD(vUpW0-F z;9HcKgZZBwJdHZqYKwxDT`^LENuCjuVidEuOu2H(apDOp4*u*eSpUi0uZ@I1>mg~Q zwbB8YaE2?r49hIbpeMDwz`E{>eTd`d^EqBdijx`7E?$L%Gu60haltpdV5`6fjB zQk$>$8Wa`ZwMc1|a!a(0-UCr2<+Dj_($Vb&d32@|9_=)TOZ7!HkoM0Q! z*`jwL2F!68HbzmOERM8kAT0}(6>>|*X`Brc9&?OE?DUFs>_KqJ+|bPnPdg|en*}*# zp9>Z;FgLqhbgRU+ND;vzmpGH+E0>y3btw;?EwrzQ)gkZdn=?G?d zawk0<5aDamvPAgQ%t;b2bohdhX7zUHLDC9YlNH1Z91}0X04X?fAwkYG`N?on12h-A zGuwUlsP~*z$&i;dc};B9J@wXWsudTM&0PhpPChO=NZ_t`yJvEm-DI0fA+{qP^vZ6` zzD_5sZu-x+qgGEmZ7W6iBIS`n$1oSlND(e1w!~(oKcTt!s{$|eyOloGjcLm$x zRBcdTeEs2#nmBWSqrA)HU%5(Q5U5JE#N$#H=}=c%OcfAruXos1It^GiKO2OBdG)4H}Y z1?l33mB@iDi-~fM8kOakCc-hkQpeY~56k};JiO1HhI&N$`Waz%nP$k>2R}B+H{{26 z1bIVRvlaTJjkCbzMsnj`bD^bCy_js5)|F{m+0SCd+E+ex0@ANx(jO^wt7h4aObsc` zQv#9R&nX`UIJ79$^$|#!?DkUkJDNuq4HN}fKw_AMi?%}F!~1Ep^;#b5D|y*y`MyP3 z_`~QH@{D5ozhi!(Q${(QQh{}!<8#?#jc3lADX>{&1cFU3YTtdaoaY$O7SZ`0Q7b0+ z-g9)17I^54a*X+|BJBr~#f#+$nFlmvBt9>|wOnK}&0G1Ovu^3XMjlf4ZOl`WYww14 ztTnj@S>TXn)xrnP`dIEe{6IxBCxs6NDuT4qZUA~mum!r19`W3X`B$+C=F0gz)nYsbfi zdp$i<#B3I9I4B|0CFIn;U`GdHCpRTXkWHLOA-ntRxg+%=JA_b4yzedgiTrvz;Ld zuLD;U4xL>`mA=PqEGLC^a#xIF$-ff}u1c=RZZB~QwhL7jV4#a_<@`qX$e%dfKDbz4 z!=&~Shg*2XT$5|;RAI7TI+b)j(`oNBkC1QOyOT8%g7xTk@iqUYcTSjcd>BTj zC)8<6`^)t0Cz@2v+ui7g$9Q4NGyHvNzQWv>%3|VuA0+obgF1)FYl_AT$~jXU(EJXE zOuw#U9TTtwl46c;sp3j?Wot*w4reDZOVWrWy%G@4o zd7*t`vumB@$|vm|cEDc1V?^Wz%c7ROJJr{H26T}t7nVp~GRHsz^A4=Zoe94Zp~yT* zxApqzy&;K1$%Y-g_+@h4c-Pm}ONu~NiU>Z_C>X=d;ErSLh22JvWh0hG8pXeoxhJ*M zL+e-#GinK!Q%<^{#M$-4PfhYmb!}BSlN)fXX@JBKcsd{k!migrht`fssETv+UX{i^ zclY_C>|@TQb2-Eo6jI4jJ?5Alhs1hS;2~f@C7%2tpMKH#@)r)i{o`JBpabg5r66(5 zHFw75N`Vux93O;{qE&j>k7_@_3>_nA?n`ViuACxWX+n9H5zJp}K>^;2$_wGaW~-L4 zM>3dI81iydrE4I?8CK3ak-K`t9u#SspXD7f%E3>nrH=Rs>s zkjf6RV_~NU{BJmcU~nVZ!0zplJIzU7-4eTeuP#Mh-x(Sz&uV)Qp$Hdf(R>b_hIdus zae2-J11n7~$4P&n;N6dP84!`Ke7%lmvy9e;( zy*ss~RM+9vKV+SVnyeUT)-WWC$0s)IxbEKWL0>f^PdB<`lm#3+mkV^sh^4E%h+Lo* zDx|hLS$qj~yrdm1@Kju-i*0U*mSGXkJ-g1cdctt}V@31vKA2LkXJ5Cf)TT);00;c; z0Xyi#cLg%h$Y$AwQiX{LOhRwB3kbjl6fUAKwr9F0ix1}Eb+XtKMZoL~Cek&iO%J^$ zOzyM_Wgy-$L_?{?GZe^Z{{IQ`|6j>s&Z_hgE5ph{ zCI|0ko~1~$E})Jzy12|uRiT71c4lR40kT=<)p!%u395z;>|nQIq&3(%XU<5XdzADa#@YBjvV_mkiH50ZQ}E z@Lc4E#E~7`E7++kpA_$(91|M45pXK1{1r{aXK~dA@q#M_CrNMX5JO9^OZ!scKZr870FYW5;`la>W?(P#4%J==} zJ`zAWe*t&r0lbXhU1Cf4?eyk(O;_Q+U^Ydlz1Aww??{mx6QX6{Hi%!)nSBTPeg`2d z)Fw6#efOUn*Sq)c4+eyWmX|L6c!2k;Z>G>Qz(X)$i_IX!5VmR$ zn7u324{VeYvTXLe<4xeVDKnnB%Z!a^T*8|&(D&n6_Hk)wYW81`iLod? zd+*KM))Ql6V?SZfexCc`Ul$!-pUk!Nt?%*nTmIh0@$q@|-{|BGCkn3x>p{&re!&Q$(&>1RV?wc~rmg!*5U zYcuOD+d%Ksg?TIVFDUE3GcE~9Nx$0K+F#`Q-J3UW`W^NR&7DeUF@P?7TLthAQj`PU z)Q*ZK%s@KDhR8Td=P;?lI%_h7b*An^`cm?n&nD`g)47(fRd?zE?WjT)i9{XjB(F0W zOx_(*)Z)vWbVn7fIBC*x?y#8XiFH5(A=|Y1&93|H+h4HWXx+HwJZsYKY-%~{Q46EV zQTG_e&svP*w`>=dN-lhhDPJgW6*+6Q47>&V`~_4{V?wE*!5L%sPa5o>yPCWXAP?~Y zDe=lkvC2Y<{qId3jINY^*S?_z)06sMJ-WY=$B9++F|f-pBi%LHss z!DU|gc}kxSi|MU=mbpJ#S~+;+>r2;8QJq5{vB4SKm8q66!LXh3ijp;kXg}IDQWs^l ze(u>|4r{u`Ip&>K|619^_;n7_nEc`0;vw4WZT87F3|qg``uOMEWem>JiX2&)zRe=L zTxkVEI_>gaJ!{p}YSYkAbB+9LH%mJ5@w35{>Bwn;+p>wvZl^e#T9F@d8S?d4@#{kH z&AX)zSk|f>9yhu#vr4xm#{u>5zSvfAJ^7V6suTy;4-Ozd zcv;{cy~8=_A!i^%!Dmpzyf+umwB8ZKcPLL_)?vhUuWz*J*q&+;7~v?xzK#ZP(evQV zicX|e4*8ilibIl!C*A0pj7#*6qK5YyjL>$(7GSF9i*U!-^$qqDqkZom7H9vtZkt48 zwiqSx5NT2F^QHCu3F}{4*+=I${J2+huF;2lvH|+zkaGMo>F}ve@iaq~!W*rJH%H{9 zAc*%JKqGf9T{PAtrc9^q{WH`U3ks&j=F4vQa^AP9W>mVEU%qcV059+;j17=kU1NA( zkmq%ei^%x};iwrDZpo*+=kTni3O2hNlv--1g zO0z6-t}BJOB|S`eaKi1b6SXalzEf;wS@BvhSU0F2fkuJ%;DenFI`o%%bMB*KZeM(T z&x{+Gqg?+fJ^tA+pfR46)9K6}WmHM&u3to>kSZpH2>2TfPkEw906!>S(LDJ_VDor zp8Az|_xJ^gqQZ+k2$~gk7tBZzx3(T=kJA8FXF?H3Te*Yl|>U!?l?%mktx( ziX;_ezP%X$gWZvn6^eytink~qsTvA5SdqgC1o}uhA}uVX-3My#{=tU`T>JPy=l$B0 z>lW9>slrVkRpl!E6MB{ad|ETij=}H^iR@kLc8|Qd(+||VhoyE07x;(0(zL^dR{(?> zUs|7_j13QMQtb-p#K~8pD0mgtlMl|-?#h%dZ%~nj3*H8Ug~4d4ya6UwSAFFH^BqbH zK9F&!jF^lx(@7$&IaX%a;sLdl?j4Xs#UAyT&MHgGgXSdY;c`kS?pu?Ms%F4?rhV%&YH;46nNSc z=WGI+2eti?SB2E~WDH7YN^5%*g-s_)E4Zl{r@WnhDI5-{9riZ!jC3d5A^O!L&=ZKI z5CHkbZHrFdBfSZ?CM|MZaElWH0B)W2JAH3SI`|xwZD`D+Diq zRiC*3FT{Gl@U|DXRvN{s!AMJQGbeQ=8s1YOhVySf*)AftW)m|G&U7;h9g3sk3|d_4 zRBOTFXfp<$*E3Naz!QWAiT-T+>}l6GR5;46t~>8lg z?zA+`Ft|{#DZ={%Awfd;wu#R(*CcYP1L^8*^_xF?hFw&#UTgSUyXPclLc$XK#BM@C zgO7+2ITGJ}P6|g9QAZmKYEVQJ)5Vfii5sCixuXW*xO9Ue#&fBgW!RdYr~_j=IN}KB)9p zN>;!4kw(VvBSyI!KD2zCN;68>v6Guif5Z($*;O$J6(IqBSvr3q9GB!c6n@kcC(iOD zuC_^UgQ6BAC#L5QERKTTsIXDT<&nb&hc9*x_=yvD3?Fu?b6K zpgRg9;&SgE4cOu+zy^VwO-=S!v2DtrBg(5Hdv8{HJUR1P_K*)by^ptgmgR? ztAlFahV&0Y82bv9y8Z;D^h z9tYQ_iIuIF)h2hvAi}#$y{U&tZr06}vL>NnXl^5Z`&(rlRNv_5ITCLyYC6&sL&1$5 zGYnd}JaVcur;>5{8LVwQA<0pGWdh=GIPR8Ho#tXnaP^p@d9KLCptKzRbcK9}D$sBd zsO??Onc_sym07Ew`EuBFm86!&vD3g`g4E-C#tQ)>AZ#aS3#^ujT5s8A>okN$gwZ^eRQcqXdqeV61i+9kc3H~`J&c1lUbod?cgLER0$QCP zK6!UyHI%b;2@k%g6#pr?`gAV++d~2A-Jt;k z$&n&!ng{C~VJ`Tl9bcCGZ%IoJ=1Xw z@l3q!+)s%)Z(_@CO3qL0Va~bK`6)ciYz@=npZ%zyySqEJEQoe;CI_Z@ksFeRAtr^5 zKoDyR89@1vHWO6$%CZEMZJ#&VfkZcCtou#BJzRW}FmGMXh2cb7;O{Cf0TBO;d9)}h zJ1STh15rgwy!QsZDw&7C@x~bkm-qm(q0%!g*%f{tBNP-EqM)om zRwzm&r>gC6LW(AWQ*+sEea=FYU!Q zMxP0;3I;Tq7TG-@q?cm!{K=hc)nMU^@ z07OLA7UZ2xp>Ntks1*T>d^`0pW;;=5H$^O1!Gv1L%vLsX+6EjD#bB^;po&)YNpPTd z*HM2rEqvfDRxOHEo3Xe-Z9u)iSYK{{op+*6bFz5bdh3%ahhL}Slk)6e7ExVnYm|r~ zt5ES+&O}Z?Fg~&waoeLlZiwjD&O}_1c^mTNRCGJ9OzNp9cW<#dT=}=F<)paZ zvnD~3L-(;msvJI*iYH{?+IranG>NU=YHS6gQW3dAV?e`Y7FUQwO?H0{&{}d<^R+6k zegLZ8w*zl|R+g%YKKtR&=D`kFg*$oI z*?JqSEhKQvrj~JHqHpr(G+X2Z5j>pojN+n^hKt6>NVi|64dkG<5<27&2|!2)_}|60 zAtQ^h-2CNu!X!Jxq+}8pStwDOttz<4Cwrw!PhXP5444E)*~cbZMu}|ckj@@iDlIRj zsUZ|YH*M`2s45sffzVOUwkVfO$c0qsBgUaodHMNVzXQuJeM~3G=&T_pRGR8C0xg>@ z9Kt@?RRm95oA^A*7HE&}kWNqWTi(o803}}CQjdXVM4@|h{!o!8!)n|JeXNP<&2>rO z4AzEQJaWhmD|UiCx}fl_Py=&*Q%Lm}fcf2Uo-KMTxUVp82#F&uui4XHuq{y{eET($ zdDR-@DUZdl`x1r{GChG(AmLaDYuo~;-f?@Q7rqb==h9JFCX0r*DE3b{qf_$MJ?xke zsA`x2ia*Tr(L`#4!Dh&3E!^`I6fE*B6-s-&OG+2VJqDgn?k?o&ga&IywRwvu@j_P; z*z+BKz63r}>zTM!B=8tev8eyXwt63N(W?h%e*b6ux9W#xciq^Rnng~lc#kN%`E6J! zLM3_H#oJ$sE%waXMFkr?O{@+MKp3!Njtd2WifzQ=&QNBW=mju%9zmG*clo7QLHfZ>yY!y zEiVD^GbI&wj^@)5#Mk?7cA;lV7l!`!a(?WwKcg1{e0fYL?j`SAzO(A+2?G!%4u(&? zlCC_`finOS5q@6g)1M4Ms{!8PhLO&8&{Bu-To&F`cxjGpiK=C*fH5et(n|qU=ZvZ3Fa|!|TX3+U92lap zH{T2qG)pFATK}i4H_i=KVd!On?N;5ATN1u&q2KGZ1o>{jG0yzkc3=wL9wfb(*7&32-1My(O#oGwXDSg@z=+w{+5|Fimy{s=(?usX0?=c}Me|EW(&S^=eC z@}GJX>UIF@0m{4lD65maB3YXEq2e;L-@bg?N|Q{4$DIEPlK&j;;Slt2M@3oixJng$ z+n4en+S7^4O{MgK6>;#gGux?V8*6D$=^+mZ~0D(Gys5=f~x3m?!onlO({itnV3s0-&R-`{!4Aw7G&l?%2t4Vr0 zmwej3AvA(gcY9mT@(9kpOvjry;ssnsMWu;{8=h^QkuT zW{rMqc%5pOPfbU@UksmlGxAhBDEa}|XpRx|&T`UISWepaxOuh6fZ>&J{#6mb_W>lp zbJZ$+XZ2JYf8z5I#g22nzT2fvdosx3%;ullt9Q~1zPNA8 zk`L6^cMp*wvRpo>ZZ;X+1<+Q`C;3r-?8l|v@o=XB3CTQM+eFyYyFADXverLJ;Bj>Z z1PREp6S1>_He8N zIHU5H26j@TOjicVBQ&vU1N*fpBe1+`^A}9KkiM+Rod8x9(eef^NNBY##ivRY%jnM< z2I~8;Udo1i>d_W}WwwR@2CHCA^FdJGY6g~w%=3Le=?0XI!kiC+Irv_2*g8|3ICP!N zTdXc%Qohb;+q~t=+diASe{d!26Xl3W#4mvw{4IPFJG(V`@O(dECmwo0p9LDImGmvqt4g=!Gb;q z6o6p}+O_!Lzg*|cl55LwZ7mmg5g7DUcxBlO(cpmifWXVR&dfqj_(19t6v z=n2*ft;WOq!A!$OmH>>2k!JXupv+JDB#p-)5EcDhAQK@Jo@p^Leas}*i{uDI%MnH& z-{i25-?Q$;wG8Kd`3gYBj2t5ej0e7vb>5ivWmEa^ys=cG`yOw%w89XIa2!M`H*awtp`ZkGEzX5hGzs}9bZ#D@-4-_K$sxDU3L!x zJw+WKEf|lE$>=%2HN=Fw?yUaUlfHLvDcbS`PH$6(kn^5UV@>R_x}qmL3MxAb^{WpB z6XTqP?4p}crcLe?#L@FWSI{56J>dtx#qjpPiNQIz%ISNM6@lmOzeG=SNYy$VkIVAa zTlt>*fgq!>(H40tUb#JfnNz*wt$1mqR?BT|T~zo?xnn5jRih?=00C0pzyxe%Tc%VP zXPSHgE9!Hn1I1;(tUB4Wstzp;c4HYzeBA}<=W%QfZo_guFUZMpX2F^y`weDIt4t@s zPKN!WsO}^bu^4_!rn@yDj1?I z!&750)wkj1f=@kB;RCZJ4ia;el7Xoly_tO|;&+!vPbj9jJotQ`pUw_CU}^C25#yI{AKtROir>e2%hlWRx}he)ACh#WmM3t^)$itgnHo(|xj42dl%?p= z0&qW-B1I?JhIDNDt^$*ntx$Et&-q~a=f~Zk2r*mQnB$;qX?Io_-X1?Kp3dL%x>Q+1 zYHL#Ou_jHt0T#Dzo9FL5v~P}eI3p$cAJ!gtc4wQT5UE@CA1|!yU)8L-HniB}p z;oquT!^0=C9NdraN+A+>-e@;VoL$)CGs9k(O$=U${{xfAQ&gx-dp2(i@&`F+%k3GS zgZw=2&*MYL&h?sYcmBHeFvkPtZ9lH%a*z~(SRDdwhvDsT>2)3+fcD(9W!otIY!$^t zivW}gARI07QQNl_xFE*3YXGQ=KF+z$t1Zm&+0fHVLec$TZ$rtM&W%TYZVDJ!T-JWx zwxP7e2SNuSAVhXgAd6Ieg%B%eY9cLMbwnyb*&&?W3bqV)+^ZdGFD}~`BlR??Hr?-% zqx_B{6L!!=SHruay{N95Yr{SD30WcG=KFV}IHBWVt?4^2rx|KueKquH(jhU77;=8Q)yT0`KxQ=oIUc(Im}g_QDd_9im?p{B}B!o&2}B7bT0z!3@BF3 z1c^~@RWDV{bb#%xIm~S*D7|LR$Jvv{{oT zeX-v(yTeFFclXQ7xdZq>yrXmNn8bYS{ zOY3S0r=IG@0p`W3M(qzRf~!|Vftq!2*mO^Vr$Sewtcilx2eM}fK*Ff|4CVy5HPzmZ zgg0!pvIcqH|(s(hQBr>1wj@ZhP^+koZzp5`Q>jdy=Kp0eNM|mn$6TpDY{%*T^Jt z^2`-vNn=rUF_zv3tTJTmaxCrLwjO<;k~6;RWt{V*(H-!FA{@Wo=xQQngmV%5TYLkS zE0s=LZ*A`m5dGP7{n|qKqYJU6L1J6?2yIxkZghDf{&SGhnagIU(yFbm@sR&n$OJOt zYMcL~+QS?}a*dyP*C@K_@fsb&)jtQ#ZU=bfgP(MEmO(X7N$18nbCSiLxhV*ct^}{& z{!GVX|4FfKM+LR@_BaCk6aDC~H4BR>YFJ%FqUZ^A!B0$ti=|mK2 zBo_|VD6UU;p4Ah&4L{Kd0=5wl3J(7zmUb)~s#@G=$_=B($xAGJttD|CP+0+r0Wu~C zvfxE)QJTQw)trT>r4aZaE(%p&`tJh{>OxJ3xe0QPndy6U0o*Lm2jRp3)|d$iQ-mP; z?0tv@GQYviEGtKpi5Ot9nep*rOx|ODlpjD61({LMCLiqnL6d$cHuZy8raS9+Y%-`} z*x5fdG9MlZ82&=>Ae0uCegfoDsI&99o01?BSBitpsfji`lzEQgh6i+3T~tqpP-v;$ zS4YBJ!AT0CXDUbq!)(n(5bf%%*4ya01Mt{JB2Po6MmqbG z*&zSXi;j)|9f~QH94gU;uiJ|I?)V&hfNM)~g25FamTn`&$Ac2+HMJ3z&ErO~X2TEz zns>HCGW&oc^8AqoocB@ezImEeHp9~EPZ78L?4UV32le`Tt z^AUhW0c7FF3PM?{gt zIDQL!s`CLf>5STKaO8nGX$T}`-3kEn)zCZzD`v_HM#El5v%UW>T2&)Wmg~S1fgc= z!^8aPYeW+3BFxmr$>IRHlvyL1`^wG&hD`ce)h0X|UVYPjH517f{R??<|u;mQJ6 z$*AV1fC>Ihj{nw_E(*9D*r_;PJfbfN(Xy|3e^a3-x4C7fk1xM#&|j+N-=^zu1TmCm zOs{v~c>A5voLPz=jsKfz9W=HRKcWS|Ylc<@`M*ajn^{wXZ)=K7S@k@4;xruuJfC*E zY_r{a$h_05qw0t`pnC?hX^$&#ZRvn3_T3o$BQn{KA50kC_9}A_!DS(m5vcIsPqhGw zfl6OV$%pCK1X=raBBrHXM>evVuXYx`jm{xBm%IddrHMmAr4AuEWqNU9Z;3%5;ai+# zM|@@in3z-@z$WAEv6dzRq{a@8yi9dq><~a?`C1{{jTq0rq>IRz=!4w%y8skhDF{Hg zRVjH$L?+CH6~G9zW&lcucRDJ~6ua}DW50LPgWv5;6jU8IhY zkznhbDku*iJ0SArXL=vl4J*w`y7OurB2BMROH4QGg-+NoBkfsS1qZHZ`FB^udLxJN zr4JLdD)jEB%O?+J$eP?(BLli02k(5St1s6!y36GVNkB1#DB|K~(+sks8K&1#MDa)| z2d~EHP4rxXlIvBu=sxJlO{ZuPr%mY9YaLG1m;GYGt$_(w8#f*ATD`#`e7Z;685bv# zfs!>KTE)S0S^K3CSGFP3*8`#)tNAwEsUCfZgrj#gnIgivHrk~v-P>Sh)r4_`5~Pp1 z(5@3-P{8FuY`xo$+hvLjbEmpuh*PvKK>h;MUIWqIcI6EcR1NH+4%|u0LVk>Pq?GDx~(5h6Z*qcWm>mpWVVcrz+d!Q6sTbpIS~|%T3>#26#u-1BZu$#B`|7j z0K)LiI6<)U@ZE%x93q=ZcPaBIdHvJU^D?zKIEZL8ib;;cvJFicre(k|91Zh}n;E0i z9%#qR3~bhZx4t$LY`VW&wnpcW?2JnO2rxd^Mg*>Y11P#C83kvfV|E9Qd@WeCG{#}J zs88wAeO}iE9oQfZVxSHi+Ve1`KOK;*3u&zdLI)L`Ynn3=IV#5TK2#&Po~E7%ySoEWf{H zTm0I@%eS|@-45MfD&LzfVnn|pUTb2_fM|B_xlsmU}qDM`u&`C zwBiB%@@V16!N+@kwn+a*?lO-V^&i$k{T|t+4Q&g0v7O(TA4>-b)aybg z7~saG2tfy;K)ZpEr9fFB+B9_gM2kaC@6xwNvcu?dm{3*C&1mwfnk}`Pz;SQ-YBQPB zZ0^P@9KT-=Mbvv{cME>6C;xpPMt zc|jXm4UnAsgG23-h5`GS0Jc?>ffHZQ9hJ}qt@rsY4m!L^|9PnBLa440FG%tIR+V%S z(9+PLcyyGPm~YR+pUDV>5J`|xs@KKR6ab8lF}gzs(4YhNiIh(0^WbiEu%#rF(b_nf z9B}tYX+Fm;O&co4E)kHhtLz$tv$R0?fCM~MCV_lDGBLCe;}rVvvL|pyOX>x zng%w2YDyV!YnXUp_BL=P-!o}owqaf>pjc@3XDX2*lCvnfdXM$e2y#u#RD}FnS_>dN zx!%0dA&tRQs@V^LJzf`@rlZIcSfh0A_6I5CMaL^CK9TWjan3d5G>fO>#8}R1m6xSi z?#+s*rm^$SuFlqH+9>`)?l!hDzFLf`(%RZ4)IK(S!|LYI*;39Dlu%+b(v7LcX8GUs z3|0O>ygOp4kn>@BUs?HPcoy+UT`F|&q|jz`Uxr-bal@RBrE<@58~fJ6MkiZW|E^#C zvN0MN{2)i#Mv##$%2IInJAWBvxq$%HGMHgk=?fsp;;j*;BNlWfzvv6{`ob4 ze}2eQP`rKqIeLmF9*j|YN^it(V1?uStc$NZwX09k#1lQa^z8~6+-#y zJBD19KHl`&Z8)X;;d8;`$w&?|q$*@pcKH-^*hO;X$@ZQYMbEy;JC>DcG(oEQ)RrUV z1mg8$eJ(2n-7QM(2cWsC2@h{~3t@$^!O=O~iivxiqKTF>r!<^$_#4IoZjS+U2--Bt ze`sdYXi`2Xfh@`e=a}uJym8mo6~Ygod`~=rxAx-T4I%xX1Un1C-VNQ>-VxR_kJ0)I zue7VW*m4}r?FRMfBOH2$uU%yLn)hI`>~j1D;6-7h#DVbe#G>6E zPN}15jnSj zEvGqn?lgjcWhHROeC<2^j!Wf!waWO`0~}p()sg`~PhZ*0etSLzqt9UY)r+-PPEq|3 zSmObyaQX*k7j@tZM07Kpo@)ZgPuu~Z%zsAvoS=Fw;H(gWrqH_J z@Y3?Y2T9o>D9J9(i(^}Ll-*jOW1|A3!^Yp&h|)M?848|uH*fYQ>^rWJt6)UTIsD~1 z*e1o$8LuX%FSfNjiM`*K)73h8jm>I)SL{H-kuL_lE5xm9KUxQEVdo2iph94tTolo% z85V*O^2s;SDwwpNZra>cgbNZ( z3aoXm>A_75V!G*qiYT`r{l%|tJd;EMTw7}|1&?f^ktUxq>y5tYklH)-FgwcxU@4Ls zLcp1%&))GY!;dAA>`>V7ASKov#3^RojUVuM>S^}e8H>VdZI>#|dlikecd;TBjL~Ri z*3j1bpH1UPqW1=N;N@5ntKl)ul|m*56o&IG+mcz!273$GX<%3I%qh2L>)0GBAem2- zF*zPXN-d$@ciR~+tjwV;^@o+vn4YOyW45$6&K?5`U5tNKtnWT&iP@|%|IRim_!G)2 z|NB3lN2i9;72W8C5acp%uk>uc0fpC0gBK(G)zvS@3HVZlNr!I0=tOacINPp{7X<=eJ*O1}5f5VPN(s2TgQ9v}k=Ipe&$G3?*LcgJqc? z4}mkQC<*G|DJY!4stW z-X*-vSZLk{5ZYHwIVSWGM2tMJ*$N0EAdVe7nC}`)P6{x=i~5-I2*3^*@||@1X#F)M z{sPbz@>w`vqL)TSamX2qskgUTELonI0-elZPakZ{A8cxIKvh`pI0Zrd65UXpU>2XA zM;NnyI9I))X^QIIkp{?Jd!&FKAa6i{GcOGEb`EL^1#F1fIT0FzVE)-))$}gj?CwL{ zIqi8W2oWaLXa@BXvO?V{Je*)7szZ4F%wZgw5@08v-d!gN-c3Jv`;Z8B-jEf#5d*2+ zeQ!ev9#Z$~6$tg!@yc^f8i0N`C+V2M?g6t!BK~`60BU@O#bOBtH&g4FofNgrFOEa( zVP5fMcED|S=0flPyuo=)dfo!@dLQLB@Aw2z_P7sKL||v@{NaU>CcOn!Sa49 zug_=z)Wj^y34Ax(;B1j{_8nNdZ-qo4G0;MSwTH@+fU&%sU84KbDHa+U8dmJ(kCNsd zZ2|njucxKY$M)y7yMr#rp8~e7&uH5>hR3Y#fsW5|tY*E&e`Zkg@c7yGc&*Q~ax;$v zW{h_>3HWc$@iB8aXY&s`5e@IPqW_-5iGDGB-}L2QKE&s*mk50Ns}}u;#{TEa|2aqepZxWS|0Ab?|Ld#bbyBv0 zPdd~~)9KqE!kGDc7)_j}GHPWAgDAf_0gf8sLwQ+u-}&*^GN2!g8H2MEsvn27xq0(W nFM84WZ~SqNdDC+;XMltF7-@>6S&ZC<6w+X?$sgHz$IkyROXu)& literal 19821 zcmeHvc|cP6`nNTUP0eUCQ`u&WEmKx5ZCY-W)tF0CSedy{nVI{PBBBD7m6c7DsaU48 zh=v=vlAut{lo~FjC@LbAfdL{33bDTj8?Ao#z3*@Cy}x(Ae{|v;&w0-G`+T4E^E}^k zVVAetchl!jH!?E%&V9!=sFBeZaNwWSw6B3H8$0$>fd8h%Lfu@A%32o+fRisnx9r?v zWK@owp*i#waQ@A&JN#mejOHve{F~wqUHZ<*Xy;V-ZCmyv1c^B@_W}f{#9NV>hArI$H zO-E=Sx3quRvUDA-U-Rf#Vvab^jP{NEb?)fmCw3mBq+IwVx4S*eGnG!Pv3uCd-g|Q? zdkck{yg}KilxVHvAQR8k=8=kq>qF-QBYg}N0>`O7fBCo|K3zwyAgv$84gB9-R6YLrG~K|L}WPBcoV+>?%OTwcKDZWl0$_@EB#`XQXaqg`nr~4 z63qdua2xKc?YQZPC`0O$gE?x2q^*rZle^^e6d1i?5LY)i#)4rIl-+8dAI4uE8Y5wH z9vmOYUmaL`A_U!veIp|UQtiN@u_)0{7RA5_2~9n$Y45~UW#NgH^4EX9i%=$6l5M)k zDItF|Pn3^!LF;6Au%(|s1$zI`c#$QFtCN#+ihW6W0bR5@b+ms_yS$e$tQr$7A2g8M z{*K?0gUzIjmN(bgAl*gCf#2*G4ND5aT0ujX-;9ZuSPRf!uA@6e2PeZ*X8{y4r?(?zC@?I1GL-GQN0W_6n{us|o*h^v`!@agNGLFmD< zKw-PhTK8pTNjVyZ*NG-ivODNiBq_AB!N%t*gY3>~5_z31Q4b%YttJ(tDD`x_b|@5L zO}hGlQ7kCSdJ9bR+6+Q#mYaX5#Hl2a*2=dgr67hojrQMK9O7P$@U-h4peaH zb@UAeF+Yfq9}uPoJdoNf4Z{Q+-$4}7d|08tKnr8tMd-S9F*aIFkV7sB7mep;`UugZ zb8mt%K?b2R?@1yu2$r~f&G3P>`Am2W9+!-4N7=``poWV@ys6!sAWG|<_I*i2PX$>2 z_=a3I&B*9dsDxnM1a9lHKHPK#ipxi|wL79}ufSQE(Xz)gY;NAfvpUJcnitPP*b#Ie zgrJ>pjTa13?~fIAl@@shQlFbITGZMGn3Jb=7ocYDI>$sos1X;f7V~qs8ov9XlJdro z>E=d8Ls%&JoW?s~r45Imwhp$MWl#(oLiA1)Xv_;l*F3eSt)c(oJboEWxa_n0Ex?NT z84g_q&!-qYj9URO*?t?CW7uxtTra~90q|qn_!X#dzx((V;D^Wf6(8aL9phIPlrmAH zcR^9KcV}((Pqpza3m7N+`WH($#CeSAKJWe?FvA=hb_)M=xzqwy0K}j6HS$xpH z(AM>=vcMdCiF}F}!}THP^8xRW7Xrt1DYP zZR(lH0}Hs>FqtzWTK2>SeH>jn^3Co2cGpROL*l*JAQu*sTHBBTd4-eCtB(`W)?n=N z?o_5)`tnf<|EyWi##$XsQQwkWegQt{+Ta&!x+ z8s%ZWx{z7^V#{FIo0#UX9^W0jFP`sd3_>e;sh-p-c*crOe1TvS4g;cym|>oKe~T|seN=`LP8+nGPJ}|a`mWFm_D@m zM~Ksly7#ZsmixU2Jg4TBQ=xW;5C z+88Ba-(kOzA>apBo;loegmzA2in4a2X5Y0SSey4*Ij-=MCc2DVfOrYb&;Kzo-9gk& zwUh`!3CkIDtvCi;mwZY8W{>dTCc@?1EXu;U#JOTA21MN#<(TT`c+8CRvvJ_cqL})z zE=}LaG=SH+q!8n{ttQjRVLOPPvU1^1>)U))-BN1>W*VS7-s>d0k2x!(MgRAJn z`(_hMoP)#WwDSaUH?VDJd8I7Ed1d`Mj6{K!yn=V6qg@KLl>}udhT_>Dp9ITl@l6T> z3^)1L_;v?rL*1OHbzxj?9H^Zopy`_MnkLPB5upFC*UJ`#JWbVTVC{%{i`X0Tkq*3W zv?)X>jYki{^P#L(Wr!ka_P9WL&vq@$r1g+ZUrP16RxQ#+2N9E z79Q>X97osi_%gg+(=^6rdx1(-17O|2HCcy^D^gs|@-E9Uoz9lMu#QHBSOwaM8}5jB z>MP`qemwYn3OM1-=JhapD}i)$or_?}Ee~_`uQ!lb+X05Wyx4~2+Y*LOwGGUx$(1_l z3%!FNN#F&ueTocN%Z zMb*l;HQT2>srXGPblh=Z@A*0Pd}?JE;(7LHOJfC697J~n`>C+GbunqfTNPkTdYx`q zHHX9PSPO8sO6D>)uWb;aAhcCL$;Z6a(bw|tC_72$u{R>PL@|?}o4x`*RKm2Ho+aNB z#J-dAIDdhWD8ODYqq1yRERAYMa+ZnpVPIXkHNZ(bezD|M$cu=C7pq6jf8xv9F&-Ko zOcx@Pvtz&kG-XRno(6(U$`NHyXd=E$3eOVzAQo`gM;S-;>j6u>QCtT>J(VeQvM97o zVz$IZ)F-(;ZvDI6i4mBN7LcIPQTd5nExQ!G^F410<^)~Bhcj~!=jxWN55?|D90o%If(o% z-{=^g4)8hTD=S!5mwnYVzF6_qs~W(7*KC3*2O%9X$s$@oCEPT*S7H*U6?1h_v5H}i zlE|y9JLV!)b+p$9a#hmi0$7O4A9vJA2l#0_BON3@c_U6aXt0{mPHvL?yn6b}s2>3#h^6(CWZU(p?hl`Ws5e__DYDVy4Wkh)GH#l%Y&j^IfM{>O z#FJLvV^*`c^(8TQDN>*kFKY?BAxb`pv!c)oLulQ96v$k7^B~fjg$y4)0lf2h>m%CO zvSNN!eSZ~4?jTeAHmMj*Ke-nIs~O#Haz&nQvao~u zC|utbQuEd>Y=Pwpwaa`vgYmH&jOf&7EHB*j&K(EW+a7ngFPw94sdy`QO5YtQZnbLB z{bTrc#~EQ|?W_5)d!uZ+mTjsa)#vRsP7?tNe7FONmtG1LaO$hdG!68Qij~Hx&0&eY z)><(|?qH@J8L62xk2Q(UYj=tkmPDx^Mn5~VYqd`by?ymbtRD_4FW}#QV=?Qz@?!M| zHd&1u-u0nvc~Px5HwAxFmHhs9P#!qRD_41uoNeRHsB7?Vb|mZ)9@A;s^mWQ=n7aCP zR+^cO9at{|bCdku@a!=%`y6A_Ma_8Ug4X0G2R=l?bH+X8#mt}-*!#z3{KAt_Yhk-g zWhNbk%-yRkz8qNdPn?ycq@?iQzCDJX-u<)et1(-NEf9k}xzTb|Mmia_Zi3(x0M064 zetO(p^84?<^WVQu`20Z+(iZyZc~1|~ZY|=Q7eI)2_q$Ge8vL)H85TKKRj?4uLp|2AE`yFWEGwPj>v!?_xq?Nd{%^AP78f%P0Iv%0N0s0j_ynz-O5c& zO>+hY20njqRsGl6=||_LDo-b7BeGoxmk)nqF&F!q!S$<_e!Odc;E2CxUahK7dEAKaJtooaTot;t8>`N7q8(Z7D? zuyNzD%a<>IF6AXXk#_p&-FB1Yjf^%Q^Z@XVkH;qtPEL`RvDm@S)T3l{{^+XgLkm9& z&$2TIo&NPR_^-b{UubUL0-3@-G9iX{_gD5r>So`Mnj{ZET@MY=eb`asXFox1f9qjx@r^~d zP30FM0JX0Pm#>_eIfUyuwYJe`MGDkyhwC&sV`%fdHqPw0I9s*w0w-%4jayT$xIedcNH zTE|^qPf&VF`1i(pL!?#uMc9|yJxFN>j5O{I{via z>cn%^I|wh8gZP+XE*%UWbf&KO7Dy3~0mfR^ctYG;tt6<|D67Nol9rTU`5aQA~ z)SYQ<8EMx(qwz-CWwo0)l0DcZ?k3>YmWbs zV;_?@>*474e+*NTIM~&E*KWn`r}5ta;q07Jdj^03?`}Rj+FzgKNv;p)_>nhS0ag)3 zYrWGD*~#lELhTV|JA5QMdbL3|*alubi-Qf~AAVtNE?%5%1KZa%Clk9JME7D3`^OxB zz>o*_-B;B(AHi1+i&G=l!9qKOYu-waG`h$+_Srvtq!zG9>Sk*;J7fE@0{`j2fNg^6 z-tRahSOp^Hzd_ulNVdThOZ@h1WLd$t$~TGn(T#S#fuW5KcF2H3D<_0yTU4*ah;Zk* zZXOj#rR%9()}hEIvfn--MH0m8;G|!RtYnt<4o2b$!&TT-US}ud;@(cQ?_0T;jji{u zE;>0{f0wx-pSi5IF+U3K>AjZk2u~ZTWai3`pN&$#GR1V)$yyy{KEF?{Wbd2w;{e3! z=CN;W)veX5%!tWxteiN{!1Oc^>tDrV^3uY5d8u*A9}u5)V41w_HD3T{#Ze`} z$d)>&FCJ1mgyqE0lt*Plm2G8x@*-5L>S9MxR9Ry4Ij4dq!>V;>?h#5Oe$73xd_s?n z5+GD&sn75P=e{(O2v)(7e)~SX0x(Y6?)&MLc~IbrHR6M=wbwizSOQY+kd1ag@O4M6 z)P6xp;%a5(6y{qTMu|(J+wbr4&qMOB_fI89y7Pwbg0%Ob#$WB=8fQ2>?%9`6H`?ym zGA3d{bg3N`_+{FtWw(L^oI5nhcUfauqijF*MfM$D3P^L%fLfRGh)`oS!Jy$&Pb%^c zvwZt0p4PF~CzjCYrE{@IAh|hY!Vav2sNU}4g36BGXa{wA8(&mnuR#2`pzlJ#Qqb8L z{cc1}@NT029{$=1ZfAzrPmahCbfM@9*k1N|;sFet4JJri%C4}Uko9(dDt(dByUYu1)7*HR9x+iou`U zC0`QM*K8n|nx&-B#(|1RP=#C@SOeq02_cOU(H--4^r{~;&n2}Chj6qz)bR$ZXDN4| zFzLKTV~qHVx@gN|u;myItqx^$(S*yEv-&b$h?J%df4*G-aH_%NBAN z$!6XO*c<_y8^@NPLLl(z7HqFJ2m(aGhN)?BHvUICYt+LXMGYlMAYW%|l=@1|3-qOQ zIck0dY%3@w^4h${bJ4NaSVy@c)QjK;$LzXoCYN^!XmgEcpxVeTOn>+?Eu~3g@{7Yy zV%-nerBO@jmvp)q1u0IR=!{oa&2XpT877fb(Si>{8Fm>JcU*}wv4kTzIP1Uzs=3vr zz49xAw_?o2qLRb_RbB_Jfr!GS4zVbt+Pc<`=FN6-ImN@UcCxNalc-m*=(_IM=bc*Y zRJq$HtjQ2VaaE?OUKq*Y)=p!(@-Bk2jC+*x#n6$)Q*qoH85+-jh-^gp<;P&C>gQz0 z)|%_`%B0QxitK_xD^8iZ&$=R?wD*l9Fbuvq2hh8R*X*ee zZ0E)n<7YV;l%2z#H1Qq0xu;4V)@av(wq6D2z~dChJ6&vg<7vq+Vz0bWGj-jE6N4xu z>~^dt4V45^$;lF(HJh|u^fqZ>Z#%pz_f$zemQe$+gBNLxp7FnMzU7Yv3T-kuj-Rtfr*7jUf5V%7Y0)z@rXk!u?%Za zDhkXG48!zF`pRLGYXDItu!^vEep`lFxEiykgc=8qpL6O?U)w+h4B6$R>zEjWg;Dp- z2{U_;u2Yfp6eX3erYbFeS~;OzkI)3{nyDibE^(TX-A7mR@oA%ju@i4Ut^F8a-<$M8 z>c1Yz{pA>y_2|)-*tod$;IFn#thPlrX*>fs?=_*0e<%1f%9)H*hgh#(9g0m(cKVFY zUk``>`phQNzfalQaL#kUp6WeoE)_aQH7X@&CF&|9>zu(j-=K z-eY3@@59EEt(hO=iI1ZCcX-`RnuIRShx>1Oe-)dYk5mZ?A(y8V-ZzvpX>PkSM=8F- zX>P<6LsD`sszqw3g$2;j^|eWg;iXwX+k|S-_Ps!!)w^%nCflU#0x4$3jY5E#3xJ%{<2#Gsvi-#l1u#aW}_kKUq9V;{A2R@Ipx2=^RqouP6~jm+I@Wajcm<hRrI$!%|Pm#}T!p>;N$>0@Q5bq~c)zGcNI6yr%>3u-GN?0uO?3bxTBz!;@0)q&Q zzPmtO?cwz2zyv3!KF#pp$lAc%J~{@2ZnYIH9Ewl-quclykr|gJux5C1<>Xh7QjA;9 z&M>5UyQc=7J|b`(m%+IUoJ{f7Q)SvQ2eC(G6 z%SvU&;P;Q*%a7slBrYuqZU4p)0*fFlk#6+?MQ|#L<=&piW(n9q>E!Dq7TUL1ogK%u z@SIb>fP75NsKMn&1=J;!E8B6jxD#OYuzx>Ru6^sF&a;WZ$ zt!6juRmAkj5I4s74R^FR+;P0M>>y114ffLD`h6GN$CvI_q~IYP_3nC>oa!hzX=ipb z+M==0w?2jGyHsV8!8WFe#r%|!fTDd-F`bTZWD-Z6K4U9F8-@}DU&gDLh2GxFQ3^{@ zG^SIYo(y6Sj)ma{|vue>@d6d*4mA3DK+XtZ%y?b|mlZ#6p{U((0}Z5wjHLT*+o!R5F$&m^+0G4h6Wa_@sKoY`r5R8O;KGOU6FV! zH)t7dM4s3&kl~Ve9fFX!wJQRSpcKXRur&GcPObl#rZko-6OAQt)&0R_K|gsIJr)(( z8I`^UR{RHG$Ii)san@CA-sUSfO5b`bM z4901zVwAY6`S5%Ol}9Y5YAx8JRSS#rwL3tv2wdCfdjw8S8+A7Gt;8P6hVK@`|{8-rO;!Xd$_M1*>44f$So z6`XOT*W0>H+)!-A#cEo2tB=2Z#)>kNR<%SQ5)2O%smVg6e^q}pk-_p5-3TZ8!(kU7 z6Bfp{ChmDZ+SkHLX;h8cjDM5zI(o|v;|dPE$}~~-2x;ZKiRjZ;<#$B8g zMKVlEQJMT9vy;V3(eb7#DquyzlTc^xLMR}ld2#=DExkaL2voSqZ|xlc5Mx|j$dUD8 zHUU*@KqM)JKsOvb=##~N`2w0gW}aMmo%XomyA95mpkm%6o8I01f=5H37p{n5WfRJ> z2HPU4lm9eHlYl+b68D?Dp=aUJ818;TS;43G2#lr}Wg7rV12+2p`wu_5`y`rH$$C<~ zyjQ~%L`68W*H&v!m0QBkM?+JH#@uMRzVmf6?n7G{_a=HNe4{-*+o zza6>39m{hIW9VABrdTZf5SLU*W}fZ^V$I)a($Qj=HhjD*adq&!KmSUaITiip9s;yfAzi;rvQI2O$ zDgrR-eKH*wg%n5!^g;svx(%jCQBmil2&H!})N6B<2Tob`9?jD|w$Zr89QUtrQogyi zyx})?P+p?YlHPowD4(T$t|Ktl%wvCRK`>b~)ghM#wBx&>oh>hO@wQX=9yR`Jj^3g0aF)zZYI;4n36cO7=p`m zB$(6y5k`LCnVC!v!^O1_^|M@ng|`6~T5p{OM1NbHSl+WtB{9rLcx%yz zOa=OzMTWx*MO4y>d~U>p>r9O(cuHsn>+x8@&g)}lQJO)Pw$Er zC^;NoiOQmMxRm+i14|y>>a*ekAaWa0cl|ZkJa;i5t-E9MT!D^3*P=~lPBcA;6SOZ` zwrbviU0IoFdt4}@ofM#-dHXW62l7O9SBJHo(vAZqzjgV}a+`Q8w76!|=$Zx4Z*k&a zE^TKfAd&YrL2fSv)VOnzt8$P{!So-@o!YKDe^4#em;^Y`16@@ffd>$}uNK$Z^5F8> zzsuq6cHCrH3m=F(18c>5)i=jx7w%;TjAeg^^t!fJea|2+NqMf8zyJ+(M_{~@=*L{_ zUsQoF2X;@&Jy3@l0R+o*5+6wbn4UU`qEv%)?^0U~-OAt%fN7i1Fb~t)qYb5MU4nf_ zhVFiXDtJI?S`K*v#K}3!@yQj+=1`Z)eYFPdA`DtGT>;Tb-+Jr=Y8@Sf8tYVU^6L3p z=o;lgu+D6$D)AXMEPNAk`UZmty_xB;?3E!5??)*(%1r|vlC?W;pWK$&zt=fQKJ@!w zL=ho67qAaooIP57?itp5S?crO7TCwDUU(9s*ZJt~MZ$;-M#X=)Wf%Dfp@Y+Yg=t0Y&b^=nD%Ezl~=a@iX|DcBjWBrqNE_PZYTnh}G@&kzQMY zcl7cMRu)J|cNv^<&Fn=WWux1TLOfdaW`pC|WoBvdM8=cFc$V(TLKM%Eo`$>VuD~TM z4O<{EC&th7Cm$*4jmF2%d!~rpE%2oLJ&5`JTp(_Vs3%?6nH8FcEFeF$@`u){-(u8| zOow}CpvECPYDvuRTY&-$0T&cr2BV{;m!HncN`A++hPzW|R?`MVU?zqgcLoM$Py){N;;%1v_A`kRp0JJP@`>e;a5#jlRzKT#m!(cRRD5ZFQI#V|EFk*2D?} zHIr@HzAE_u-D3J|t%zO(Y%L#v#-X?piVa}HE4W=fWJnWGC0hvxFk7^JJctmz`#)0l zSQdf9B-hO%uC`JUEK$qire}#|c5$AZqO~|nz-^?yBxX+1Fs$wlgdPdt289x;Og^lOKNZa*p{3jB#gXnLZU+MXh)vW*a$GXhkQBUa61wOVnxF8c+iS)dBf9=?z6|JA9oegWD|u zm4824A6y%m6O5zK23Ja8T&)BuJc(L0VG0+z7WcpM+j~?M&9#~f6l%C-q$@+kOrAE; zupO)CDa|v`I!-Jisl7R2i?D&aK6~E^T;_ITb@4MV>*ZqX>H_HtYe%Gc8oI!mbhqA= zcruQ=_Zi_xE|f*SyjQZ1ATspBV>REL?8u;muzRb3md(L2g)=q`rBtDCkh7m!Xux}) zq-)D*0Y9J1xE`=kD((k-biZrWki8o}9VnOho;x9!R{?QPqMx|#x38+As!SdQR0M~z zDni%?$joKLX@d>RozVjFr5Y9&^(|W@WcTke#B=R{Vgj*l`pjN0m_={uo=B<9$$4yR z@`i0IAL!n1ydN`xqxUxN@c;@Y5x$thC>(qKVlhOWOC9I;XLpnqXkIY3{oUaApH<0! zAke5RT`RweQN!#Z7^UwGg`~PHvk}bKAVAObU)B2ivi@w9dYI zJ;!~uubu-|v|7Y{4BdT%4K}ESvldNOY5r$KIiH!Ie_*bk<`w@7So>f9x=#vbpOm~l zsUw`9c2xf!pTp3r!4W~)R(dXe-FKv9?a^3Xiyn=>=t9Cs^v7CtO*R`Ox@U-M#l=B66g z3E6-n-{JLi{$ssLgf-t*c;bXOhtvv*;ui4Sh)H|$*Vc-mQ#~xJNR2*>n*Ie$w84AF>QCUVI`E+|yi zHfaKMFG0h2;TjQJUn57&JE4apaj9+F)4S@65_oF;7)a9+%x*eYw{Hc0BHa*{552S# zQN0}wwtxyn*2+W_3+VXTS6XQW8Y*P4C9jtjcXcB0=dvh!)f)*w7zdNZ1G_pJ?nk|d z-IUK96Gw27QS23Pk&E8%0D{c4qUd4DVb+{}tOXX#o0P)9Ihj{VBt{#}?^`FPE^ace z60>jeb15Oc#_BMR7;tpJLWdnA0?BdOqPFeZ?~q8n;Ne^6@VQ55XXq+F$n1FaGa)ZRO@I>|aUgOaS(tZA#G4eQ9&63Lap z5RRJmwsr>)Thi2A)NnhTh0<7Fo8;a#)6Hzq_YC8^X(lJt(H$<&y$QB>eE$>=`pA9B z!A*hv&L`W*o{T#w`eaTWOc~ezv;scHDI8X%M0BACCPvXuIW7t5u7%;4%+P7JhBRvtd>b`2c zdM|1Avq_8npBkCJQ-nY0M3?=hCnClD3uHcp#F>F2TeHW}5 zlG|U?RcYnJ24zqEJE`pd<3?U>4Uf^M(sMD@fga{glg+BOd8kYZNe70-1ik0#a;}8{ z?XjQ!H4sM$;J0cRtCpuhjj2oiY|yZn+g4_1QUeNJNfW|-dgM}|5OCt&vJ3Y5p5PE` z8=z`&c|!OOIt6D^_VCN<#=xUOzbudidO|a@n%qfw-OR}0C zKHRS#?!rkl4U!hcw~Z4h-vWofMXm)d_p*c$QVzqKYbWPu=eD&;>}NfdJ#`B>9+d+QyM=_+5R^M zREE}n2v|GX6cP18dJU|+W~c`z+AtVC5a&)qA3JOMQe(_mYD))9m&w;+RMN-1<@y$q zof%Cks>ktoM2wcv@&ZA(GhpL@0A1nlA}+ZJ>{M9~fJT87pf|l5s3RpxLXZG5PFn|C za>e<51Ll}PDBF2LjB-`Wm|hucpya3@Ns?`t>hqzs=LxWJ1oYLZi^ zr`phhu3!7<=8o}7%t7|#mJHzI0;ep2kGFh0&YCQ}POWL0*r?%F-X8Isbnx7N!Js~$ zOxI*jN947M{T7!Gk2eZEteM=B@wcB~S>c>2Eq%Pxvl From 6985f12c1c6c86b45637ea14c2120894453cbb88 Mon Sep 17 00:00:00 2001 From: Bharath Veeranna Date: Tue, 25 Nov 2025 11:30:41 -0800 Subject: [PATCH 3/3] Addressing review comments --- doc/vxlan/Vxlan_kernel_routes.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/doc/vxlan/Vxlan_kernel_routes.md b/doc/vxlan/Vxlan_kernel_routes.md index b47e02f8f9b..0710789365d 100644 --- a/doc/vxlan/Vxlan_kernel_routes.md +++ b/doc/vxlan/Vxlan_kernel_routes.md @@ -30,7 +30,7 @@ # 1 Revision | Rev | Date | Author | Change Description | |:---:|:-----------:|:------------------:|-----------------------------------| -| 0.1 | | Bharath Veeranna | Initial version | +| 0.1 | 11/25/2025 | Bharath Veeranna | Initial version | # 2 Scope @@ -99,8 +99,6 @@ Consider a usecase where SONiC has to establish BGP to both the devices: 20.0.0. However, the VM having IP 20.0.0.3 is behind a VTEP 200.200.200.3 having VNI 2000. SONiC does not have any kernel routes and interfaces configured for VNI 2000. Any traffic destined to 20.0.0.3 will be dropped in the kernel since there are no routes or interfaces configured for VxLAN 2000. -Moreover, if the CPU port is set to Egress mode in the NPU, the packets sent from the CPU are directed to the egress pipeline. Hence, the kernel has to form the packets with appropriate VxLAN headers before placing the packet in the egress pipeline. To encap the packets with VxLAN headers, the kernel should have the VxLAN interface and routes configured. - # 6 Requirements Overview ## 6.1 Functional requirements This section describes the SONiC requirements for Vxlan kernel interface and routes required for the OS to handle VxLAN encap/decap for traffic originated/destined to CPU. @@ -113,7 +111,7 @@ This section describes the SONiC requirements for Vxlan kernel interface and rou A new component called VnetMgr will be introduced that will handle kernel programming for `VNET_ROUTE_TUNNEL` endpoints. - VnetMgr should handle vxlan interface creation and deletion for routes defined in VNET_ROUTE_TUNNEL. - VnetMgr should install/delete kernel routes for the VTEP endpoints. -- VnetMgr should subscribe to CONFIG_DB changes to VNET_ROUTE_TUNNEL and update the same in APP_DB +- VnetMgr should subscribe to CONFIG_DB changes to VNET_ROUTE_TUNNEL and update the same in APPL_DB ## 6.3 CLI requirements @@ -176,7 +174,7 @@ VnetMgr is a new config manager introduced to handle the config changes for `VNE - Subscribe for config changes to `VNET_ROUTE_TUNNEL` - Handle kernel interface and route (create and delete) if the routes have `install_on_kernel` flag is set. -- Publish the routes to APP_DB +- Publish the routes to APPL_DB The diagram below shows the flow for the route creation: @@ -213,6 +211,17 @@ sudo ip route add {{prefix}} dev Vxlan_{{vnet_name}}_{{prefix}} vrf {{vnet_name} sudo ip neigh add {{prefix}} lladdr {{overlay_dmac_address}} dev Vxlan_{{vnet_name}}_{{prefix}} ``` +## 7.4 Orch Agent + +### VNetCfgRouteOrch +VNetCfgRouteOrch is an orch agent that currently subscribes to the CONFIG_DB tables: VNET_ROUTE_TUNNEL_TABLE and VNET_ROUTE_TABLE. This orch agent publishes the entries from these two tables to the APPL_DB. This orch agent is just a pass through which publishes to APPL_DB. This orch agent will be removed completely and the functionality performed by this orch agent will be handled by VnetMgr as described in the above section. + +In addition to the tasks mentioned in the previous section, VnetMgr will also do the following tasks that are currently performed by VNetCfgRouteOrch: +- Subscribe to VNET_ROUTE CONFIG_DB table and publish to APPL_DB +- Subscribe to VNET_ROUTE_TUNNEL CONFIG_DB table and publish to APPL_DB + +## VNetRouteOrch +There are no changes to VNetRouteOrch. This orch agent performs the south-bound programming of the vnet routes in the NPU. # 8 Limitations - Linux kernel allows only one vxlan interface per VNI. There can be at most one `VNET_ROUTE_TUNNEL` with a given VNI and `install_on-kernel: true`. In other words, two routes having same VNI cannot have `install_on_kernel` flag set to true.