vmx-tutorial.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge"><![endif]-->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 1.5.3">
<meta name="author" content="Ping Song">
<title>VMX tutorial</title>
<style>
@import url(https://fonts.googleapis.com/css?family=Varela+Round|Open+Sans:400italic,600italic,400,600|Ubuntu+Mono:400);
/*! normalize.css v2.1.2 | MIT License | git.io/normalize */
/* ========================================================================== HTML5 display definitions ========================================================================== */
/** Correct `block` display not defined in IE 8/9. */
article, aside, details, figcaption, figure, footer, header, hgroup, main, nav, section, summary { display: block; }

/** Correct `inline-block` display not defined in IE 8/9. */
audio, canvas, video { display: inline-block; }

/** Prevent modern browsers from displaying `audio` without controls. Remove excess height in iOS 5 devices. */
audio:not([controls]) { display: none; height: 0; }

/** Address `[hidden]` styling not present in IE 8/9. Hide the `template` element in IE, Safari, and Firefox < 22. */
[hidden], template { display: none; }

script { display: none !important; }

/* ========================================================================== Base ========================================================================== */
/** 1. Set default font family to sans-serif. 2. Prevent iOS text size adjust after orientation change, without disabling user zoom. */
html { font-family: sans-serif; /* 1 */ -ms-text-size-adjust: 100%; /* 2 */ -webkit-text-size-adjust: 100%; /* 2 */ }

/** Remove default margin. */
body { margin: 0; }

/* ========================================================================== Links ========================================================================== */
/** Remove the gray background color from active links in IE 10. */
a { background: transparent; }

/** Address `outline` inconsistency between Chrome and other browsers. */
a:focus { outline: thin dotted; }

/** Improve readability when focused and also mouse hovered in all browsers. */
a:active, a:hover { outline: 0; }

/* ========================================================================== Typography ========================================================================== */
/** Address variable `h1` font-size and margin within `section` and `article` contexts in Firefox 4+, Safari 5, and Chrome. */
h1 { font-size: 2em; margin: 0.67em 0; }

/** Address styling not present in IE 8/9, Safari 5, and Chrome. */
abbr[title] { border-bottom: 1px dotted; }

/** Address style set to `bolder` in Firefox 4+, Safari 5, and Chrome. */
b, strong { font-weight: bold; }

/** Address styling not present in Safari 5 and Chrome. */
dfn { font-style: italic; }

/** Address differences between Firefox and other browsers. */
hr { -moz-box-sizing: content-box; box-sizing: content-box; height: 0; }

/** Address styling not present in IE 8/9. */
mark { background: #ff0; color: #000; }

/** Correct font family set oddly in Safari 5 and Chrome. */
code, kbd, pre, samp { font-family: monospace, serif; font-size: 1em; }

/** Improve readability of pre-formatted text in all browsers. */
pre { white-space: pre-wrap; }

/** Set consistent quote types. */
q { quotes: "\201C" "\201D" "\2018" "\2019"; }

/** Address inconsistent and variable font size in all browsers. */
small { font-size: 80%; }

/** Prevent `sub` and `sup` affecting `line-height` in all browsers. */
sub, sup { font-size: 75%; line-height: 0; position: relative; vertical-align: baseline; }

sup { top: -0.5em; }

sub { bottom: -0.25em; }

/* ========================================================================== Embedded content ========================================================================== */
/** Remove border when inside `a` element in IE 8/9. */
img { border: 0; }

/** Correct overflow displayed oddly in IE 9. */
svg:not(:root) { overflow: hidden; }

/* ========================================================================== Figures ========================================================================== */
/** Address margin not present in IE 8/9 and Safari 5. */
figure { margin: 0; }

/* ========================================================================== Forms ========================================================================== */
/** Define consistent border, margin, and padding. */
fieldset { border: 1px solid #c0c0c0; margin: 0 2px; padding: 0.35em 0.625em 0.75em; }

/** 1. Correct `color` not being inherited in IE 8/9. 2. Remove padding so people aren't caught out if they zero out fieldsets. */
legend { border: 0; /* 1 */ padding: 0; /* 2 */ }

/** 1. Correct font family not being inherited in all browsers. 2. Correct font size not being inherited in all browsers. 3. Address margins set differently in Firefox 4+, Safari 5, and Chrome. */
button, input, select, textarea { font-family: inherit; /* 1 */ font-size: 100%; /* 2 */ margin: 0; /* 3 */ }

/** Address Firefox 4+ setting `line-height` on `input` using `!important` in the UA stylesheet. */
button, input { line-height: normal; }

/** Address inconsistent `text-transform` inheritance for `button` and `select`. All other form control elements do not inherit `text-transform` values. Correct `button` style inheritance in Chrome, Safari 5+, and IE 8+. Correct `select` style inheritance in Firefox 4+ and Opera. */
button, select { text-transform: none; }

/** 1. Avoid the WebKit bug in Android 4.0.* where (2) destroys native `audio` and `video` controls. 2. Correct inability to style clickable `input` types in iOS. 3. Improve usability and consistency of cursor style between image-type `input` and others. */
button, html input[type="button"], input[type="reset"], input[type="submit"] { -webkit-appearance: button; /* 2 */ cursor: pointer; /* 3 */ }

/** Re-set default cursor for disabled elements. */
button[disabled], html input[disabled] { cursor: default; }

/** 1. Address box sizing set to `content-box` in IE 8/9. 2. Remove excess padding in IE 8/9. */
input[type="checkbox"], input[type="radio"] { box-sizing: border-box; /* 1 */ padding: 0; /* 2 */ }

/** 1. Address `appearance` set to `searchfield` in Safari 5 and Chrome. 2. Address `box-sizing` set to `border-box` in Safari 5 and Chrome (include `-moz` to future-proof). */
input[type="search"] { -webkit-appearance: textfield; /* 1 */ -moz-box-sizing: content-box; -webkit-box-sizing: content-box; /* 2 */ box-sizing: content-box; }

/** Remove inner padding and search cancel button in Safari 5 and Chrome on OS X. */
input[type="search"]::-webkit-search-cancel-button, input[type="search"]::-webkit-search-decoration { -webkit-appearance: none; }

/** Remove inner padding and border in Firefox 4+. */
button::-moz-focus-inner, input::-moz-focus-inner { border: 0; padding: 0; }

/** 1. Remove default vertical scrollbar in IE 8/9. 2. Improve readability and alignment in all browsers. */
textarea { overflow: auto; /* 1 */ vertical-align: top; /* 2 */ }

/* ========================================================================== Tables ========================================================================== */
/** Remove most spacing between table cells. */
table { border-collapse: collapse; border-spacing: 0; }

meta.foundation-mq-small { font-family: "only screen and (min-width: 768px)"; width: 768px; }

meta.foundation-mq-medium { font-family: "only screen and (min-width:1280px)"; width: 1280px; }

meta.foundation-mq-large { font-family: "only screen and (min-width:1440px)"; width: 1440px; }

*, *:before, *:after { -moz-box-sizing: border-box; -webkit-box-sizing: border-box; box-sizing: border-box; }

html, body { font-size: 100%; }

body { background: #fefdfd; color: rgba(0, 0, 0, 0.8); padding: 0; margin: 0; font-family: "Open Sans", sans-serif; font-weight: normal; font-style: normal; line-height: 1; position: relative; cursor: auto; }

a:hover { cursor: pointer; }

img, object, embed { max-width: 100%; height: auto; }

object, embed { height: 100%; }

img { -ms-interpolation-mode: bicubic; }

#map_canvas img, #map_canvas embed, #map_canvas object, .map_canvas img, .map_canvas embed, .map_canvas object { max-width: none !important; }

.left { float: left !important; }

.right { float: right !important; }

.text-left { text-align: left !important; }

.text-right { text-align: right !important; }

.text-center { text-align: center !important; }

.text-justify { text-align: justify !important; }

.hide { display: none; }

.antialiased, body { -webkit-font-smoothing: antialiased; }

img { display: inline-block; vertical-align: middle; }

textarea { height: auto; min-height: 50px; }

select { width: 100%; }

object, svg { display: inline-block; vertical-align: middle; }

.center { margin-left: auto; margin-right: auto; }

.spread { width: 100%; }

p.lead, .paragraph.lead > p, #preamble > .sectionbody > .paragraph:first-of-type p { font-size: 1.21875em; line-height: 1.6; }

.subheader, .admonitionblock td.content > .title, .audioblock > .title, .exampleblock > .title, .imageblock > .title, .listingblock > .title, .literalblock > .title, .stemblock > .title, .openblock > .title, .paragraph > .title, .quoteblock > .title, table.tableblock > .title, .verseblock > .title, .videoblock > .title, .dlist > .title, .olist > .title, .ulist > .title, .qlist > .title, .hdlist > .title { line-height: 1.25; color: #002c5e; font-weight: 300; margin-top: 0.2em; margin-bottom: 0.5em; }

/* Typography resets */
div, dl, dt, dd, ul, ol, li, h1, h2, h3, #toctitle, .sidebarblock > .content > .title, h4, h5, h6, pre, form, p, blockquote, th, td { margin: 0; padding: 0; direction: ltr; }

/* Default Link Styles */
a { color: #005580; text-decoration: underline; line-height: inherit; }
a:hover, a:focus { color: #078d71; }
a img { border: none; }

/* Default paragraph styles */
p { font-family: inherit; font-weight: normal; font-size: 1em; line-height: 1.5; margin-bottom: 1.25em; text-rendering: optimizeLegibility; }
p aside { font-size: 0.875em; line-height: 1.35; font-style: italic; }

/* Default header styles */
h1, h2, h3, #toctitle, .sidebarblock > .content > .title, h4, h5, h6 { font-family: "Varela Round", sans-serif; font-weight: 400; font-style: normal; color: #00326b; text-rendering: optimizeLegibility; margin-top: 0.8em; margin-bottom: 0.5em; line-height: 1.0625em; }
h1 small, h2 small, h3 small, #toctitle small, .sidebarblock > .content > .title small, h4 small, h5 small, h6 small { font-size: 60%; color: #057aff; line-height: 0; }

h1 { font-size: 2.125em; }

h2 { font-size: 1.6875em; }

h3, #toctitle, .sidebarblock > .content > .title { font-size: 1.375em; }

h4 { font-size: 1.125em; }

h5 { font-size: 1.125em; }

h6 { font-size: 1em; }

hr { border: solid rgba(145, 135, 84, 0.15); border-width: 1px 0 0; clear: both; margin: 1.25em 0 1.1875em; height: 0; }

/* Helpful Typography Defaults */
em, i { font-style: italic; line-height: inherit; }

strong, b { font-weight: bold; line-height: inherit; }

small { font-size: 60%; line-height: inherit; }

code { font-family: "Ubuntu Mono", "Inconsolata", monospace; font-weight: 400; color: #331d00; }

/* Lists */
ul, ol, dl { font-size: 1em; line-height: 1.5; margin-bottom: 1.25em; list-style-position: outside; font-family: inherit; }

ul, ol { margin-left: 1.5em; }
ul.no-bullet, ol.no-bullet { margin-left: 1.5em; }

/* Unordered Lists */
ul li ul, ul li ol { margin-left: 1.25em; margin-bottom: 0; font-size: 1em; /* Override nested font-size change */ }
ul.square li ul, ul.circle li ul, ul.disc li ul { list-style: inherit; }
ul.square { list-style-type: square; }
ul.circle { list-style-type: circle; }
ul.disc { list-style-type: disc; }
ul.no-bullet { list-style: none; }

/* Ordered Lists */
ol li ul, ol li ol { margin-left: 1.25em; margin-bottom: 0; }

/* Definition Lists */
dl dt { margin-bottom: 0.3125em; font-weight: bold; }
dl dd { margin-bottom: 1.25em; }

/* Abbreviations */
abbr, acronym { text-transform: uppercase; font-size: 90%; color: rgba(0, 0, 0, 0.8); border-bottom: 1px dotted #dddddd; cursor: help; }

abbr { text-transform: none; }

/* Blockquotes */
blockquote { margin: 0 0 1.25em; padding: 0.5625em 1.25em 0 1.1875em; border-left: 1px solid #dddddd; }
blockquote cite { display: block; font-size: 0.8125em; color: #666666; }
blockquote cite:before { content: "\2014 \0020"; }
blockquote cite a, blockquote cite a:visited { color: #666666; }

blockquote, blockquote p { line-height: 1.5; color: #999999; }

/* Microformats */
.vcard { display: inline-block; margin: 0 0 1.25em 0; border: 1px solid #dddddd; padding: 0.625em 0.75em; }
.vcard li { margin: 0; display: block; }
.vcard .fn { font-weight: bold; font-size: 0.9375em; }

.vevent .summary { font-weight: bold; }
.vevent abbr { cursor: auto; text-decoration: none; font-weight: bold; border: none; padding: 0 0.0625em; }

@media only screen and (min-width: 768px) { h1, h2, h3, #toctitle, .sidebarblock > .content > .title, h4, h5, h6 { line-height: 1.25; }
  h1 { font-size: 2.75em; }
  h2 { font-size: 2.3125em; }
  h3, #toctitle, .sidebarblock > .content > .title { font-size: 1.6875em; }
  h4 { font-size: 1.4375em; } }
/* Tables */
table { background: white; margin-bottom: 1.25em; border: solid 1px rgba(145, 135, 84, 0.15); }
table thead, table tfoot { background: rgba(119, 84, 22, 0.1); font-weight: bold; }
table thead tr th, table thead tr td, table tfoot tr th, table tfoot tr td { padding: 0.5em 0.625em 0.625em; font-size: inherit; color: rgba(0, 0, 0, 0.8); text-align: left; }
table tr th, table tr td { padding: 0.5625em 0.625em; font-size: inherit; color: rgba(0, 0, 0, 0.8); }
table tr.even, table tr.alt, table tr:nth-of-type(even) { background: rgba(119, 84, 22, 0.025); }
table thead tr th, table tfoot tr th, table tbody tr td, table tr td, table tfoot tr td { display: table-cell; line-height: 1.5; }

body { tab-size: 4; }

h1, h2, h3, #toctitle, .sidebarblock > .content > .title, h4, h5, h6 { line-height: 1.25; }

.clearfix:before, .clearfix:after, .float-group:before, .float-group:after { content: " "; display: table; }
.clearfix:after, .float-group:after { clear: both; }

*:not(pre) > code { font-size: 1.0625em; font-style: normal !important; letter-spacing: 0; padding: 0; line-height: 1.25; }

pre, pre > code { line-height: 1.4; color: inherit; font-family: "Liberation Mono", "Consolas", monospace; font-weight: normal; }

.keyseq { color: rgba(51, 51, 51, 0.8); }

kbd { font-family: "Ubuntu Mono", "Inconsolata", monospace; display: inline-block; color: rgba(0, 0, 0, 0.8); font-size: 0.65em; line-height: 1.45; background-color: #f7f7f7; border: 1px solid #ccc; -webkit-border-radius: 3px; border-radius: 3px; -webkit-box-shadow: 0 1px 0 rgba(0, 0, 0, 0.2), 0 0 0 0.1em white inset; box-shadow: 0 1px 0 rgba(0, 0, 0, 0.2), 0 0 0 0.1em white inset; margin: 0 0.15em; padding: 0.2em 0.5em; vertical-align: middle; position: relative; top: -0.1em; white-space: nowrap; }

.keyseq kbd:first-child { margin-left: 0; }

.keyseq kbd:last-child { margin-right: 0; }

.menuseq, .menu { color: rgba(0, 0, 0, 0.8); }

b.button:before, b.button:after { position: relative; top: -1px; font-weight: normal; }

b.button:before { content: "["; padding: 0 3px 0 2px; }

b.button:after { content: "]"; padding: 0 2px 0 3px; }

p a > code:hover { color: #1a0f00; }

#header, #content, #footnotes, #footer { width: 100%; margin-left: auto; margin-right: auto; margin-top: 0; margin-bottom: 0; max-width: 62.5em; *zoom: 1; position: relative; padding-left: 0.9375em; padding-right: 0.9375em; }
#header:before, #header:after, #content:before, #content:after, #footnotes:before, #footnotes:after, #footer:before, #footer:after { content: " "; display: table; }
#header:after, #content:after, #footnotes:after, #footer:after { clear: both; }

#content { margin-top: 1.25em; }

#content:before { content: none; }

#header > h1:first-child { color: #703f1c; margin-top: 2.25rem; margin-bottom: 0; }
#header > h1:first-child + #toc { margin-top: 8px; border-top: 1px solid rgba(145, 135, 84, 0.15); }
#header > h1:only-child, body.toc2 #header > h1:nth-last-child(2) { border-bottom: 1px solid rgba(145, 135, 84, 0.15); padding-bottom: 8px; }
#header .details { border-bottom: 1px solid rgba(145, 135, 84, 0.15); line-height: 1.45; padding-top: 0.25em; padding-bottom: 0.25em; padding-left: 0.25em; color: #666666; display: -ms-flexbox; display: -webkit-flex; display: flex; -ms-flex-flow: row wrap; -webkit-flex-flow: row wrap; flex-flow: row wrap; }
#header .details span:first-child { margin-left: -0.125em; }
#header .details span.email a { color: #999999; }
#header .details br { display: none; }
#header .details br + span:before { content: "\00a0\2013\00a0"; }
#header .details br + span.author:before { content: "\00a0\22c5\00a0"; color: #999999; }
#header .details br + span#revremark:before { content: "\00a0|\00a0"; }
#header #revnumber { text-transform: capitalize; }
#header #revnumber:after { content: "\00a0"; }

#content > h1:first-child:not([class]) { color: #703f1c; border-bottom: 1px solid rgba(145, 135, 84, 0.15); padding-bottom: 8px; margin-top: 0; padding-top: 1rem; margin-bottom: 1.25rem; }

#toc { border-bottom: 0px solid #dddddd; padding-bottom: 0.5em; }
#toc > ul { margin-left: 0.125em; }
#toc ul.sectlevel0 > li > a { font-style: italic; }
#toc ul.sectlevel0 ul.sectlevel1 { margin: 0.5em 0; }
#toc ul { font-family: "Varela Round", sans-serif; list-style-type: none; }
#toc li { line-height: 1.3334; margin-top: 0.3334em; }
#toc a { text-decoration: none; }
#toc a:active { text-decoration: underline; }

#toctitle { color: #002c5e; font-size: 1.2em; }

@media only screen and (min-width: 768px) { #toctitle { font-size: 1.375em; }
  body.toc2 { padding-left: 15em; padding-right: 0; }
  #toc.toc2 { margin-top: 0 !important; background-color: #f2f2f4; position: fixed; width: 15em; left: 0; top: 0; border-right: 1px solid #dddddd; border-top-width: 0 !important; border-bottom-width: 0 !important; z-index: 1000; padding: 1.25em 1em; height: 100%; overflow: auto; }
  #toc.toc2 #toctitle { margin-top: 0; margin-bottom: 0.8rem; font-size: 1.2em; }
  #toc.toc2 > ul { font-size: 0.9em; margin-bottom: 0; }
  #toc.toc2 ul ul { margin-left: 0; padding-left: 1em; }
  #toc.toc2 ul.sectlevel0 ul.sectlevel1 { padding-left: 0; margin-top: 0.5em; margin-bottom: 0.5em; }
  body.toc2.toc-right { padding-left: 0; padding-right: 15em; }
  body.toc2.toc-right #toc.toc2 { border-right-width: 0; border-left: 1px solid #dddddd; left: auto; right: 0; } }
@media only screen and (min-width: 1280px) { body.toc2 { padding-left: 20em; padding-right: 0; }
  #toc.toc2 { width: 20em; }
  #toc.toc2 #toctitle { font-size: 1.375em; }
  #toc.toc2 > ul { font-size: 0.95em; }
  #toc.toc2 ul ul { padding-left: 1.25em; }
  body.toc2.toc-right { padding-left: 0; padding-right: 20em; } }
#content #toc { border-style: solid; border-width: 1px; border-color: #d6d6dd; margin-bottom: 1.25em; padding: 1.25em; background: #f2f2f4; -webkit-border-radius: 6px; border-radius: 6px; }
#content #toc > :first-child { margin-top: 0; }
#content #toc > :last-child { margin-bottom: 0; }

#footer { max-width: 100%; background-color: #0b445a; padding: 1.25em; }

#footer-text { color: #fefdfd; line-height: 1.35; }

.sect1 { padding-bottom: 0.625em; }

@media only screen and (min-width: 768px) { .sect1 { padding-bottom: 1.25em; } }
.sect1 + .sect1 { border-top: 0px solid #dddddd; }

#content h1 > a.anchor, h2 > a.anchor, h3 > a.anchor, #toctitle > a.anchor, .sidebarblock > .content > .title > a.anchor, h4 > a.anchor, h5 > a.anchor, h6 > a.anchor { position: absolute; z-index: 1001; width: 1.5ex; margin-left: -1.5ex; display: block; text-decoration: none !important; visibility: hidden; text-align: center; font-weight: normal; }
#content h1 > a.anchor:before, h2 > a.anchor:before, h3 > a.anchor:before, #toctitle > a.anchor:before, .sidebarblock > .content > .title > a.anchor:before, h4 > a.anchor:before, h5 > a.anchor:before, h6 > a.anchor:before { content: "\00A7"; font-size: 0.85em; display: block; padding-top: 0.1em; }
#content h1:hover > a.anchor, #content h1 > a.anchor:hover, h2:hover > a.anchor, h2 > a.anchor:hover, h3:hover > a.anchor, #toctitle:hover > a.anchor, .sidebarblock > .content > .title:hover > a.anchor, h3 > a.anchor:hover, #toctitle > a.anchor:hover, .sidebarblock > .content > .title > a.anchor:hover, h4:hover > a.anchor, h4 > a.anchor:hover, h5:hover > a.anchor, h5 > a.anchor:hover, h6:hover > a.anchor, h6 > a.anchor:hover { visibility: visible; }
#content h1 > a.link, h2 > a.link, h3 > a.link, #toctitle > a.link, .sidebarblock > .content > .title > a.link, h4 > a.link, h5 > a.link, h6 > a.link { color: #00326b; text-decoration: none; }
#content h1 > a.link:hover, h2 > a.link:hover, h3 > a.link:hover, #toctitle > a.link:hover, .sidebarblock > .content > .title > a.link:hover, h4 > a.link:hover, h5 > a.link:hover, h6 > a.link:hover { color: #002652; }

.audioblock, .imageblock, .literalblock, .listingblock, .stemblock, .videoblock { margin-bottom: 1.25em; }

.admonitionblock td.content > .title, .audioblock > .title, .exampleblock > .title, .imageblock > .title, .listingblock > .title, .literalblock > .title, .stemblock > .title, .openblock > .title, .paragraph > .title, .quoteblock > .title, table.tableblock > .title, .verseblock > .title, .videoblock > .title, .dlist > .title, .olist > .title, .ulist > .title, .qlist > .title, .hdlist > .title { text-rendering: optimizeLegibility; text-align: left; }

table.tableblock > caption.title { white-space: nowrap; overflow: visible; max-width: 0; }

.paragraph.lead > p, #preamble > .sectionbody > .paragraph:first-of-type p { color: #703f1c; }

table.tableblock #preamble > .sectionbody > .paragraph:first-of-type p { font-size: inherit; }

.admonitionblock > table { border-collapse: separate; border: 0; background: none; width: 100%; }
.admonitionblock > table td.icon { text-align: center; width: 80px; }
.admonitionblock > table td.icon img { max-width: none; }
.admonitionblock > table td.icon .title { font-weight: bold; font-family: "Varela Round", sans-serif; text-transform: uppercase; }
.admonitionblock > table td.content { padding-left: 1.125em; padding-right: 1.25em; border-left: 1px solid rgba(145, 135, 84, 0.15); color: #666666; }
.admonitionblock > table td.content > :last-child > :last-child { margin-bottom: 0; }

.exampleblock > .content { border-style: solid; border-width: 1px; border-color: #eddbdb; margin-bottom: 1.25em; padding: 1.25em; background: #fefdfd; -webkit-border-radius: 6px; border-radius: 6px; }
.exampleblock > .content > :first-child { margin-top: 0; }
.exampleblock > .content > :last-child { margin-bottom: 0; }

.sidebarblock { border-style: solid; border-width: 1px; border-color: #d6d6dd; margin-bottom: 1.25em; padding: 1.25em; background: #f2f2f4; -webkit-border-radius: 6px; border-radius: 6px; }
.sidebarblock > :first-child { margin-top: 0; }
.sidebarblock > :last-child { margin-bottom: 0; }
.sidebarblock > .content > .title { color: #002c5e; margin-top: 0; }

.exampleblock > .content > :last-child > :last-child, .exampleblock > .content .olist > ol > li:last-child > :last-child, .exampleblock > .content .ulist > ul > li:last-child > :last-child, .exampleblock > .content .qlist > ol > li:last-child > :last-child, .sidebarblock > .content > :last-child > :last-child, .sidebarblock > .content .olist > ol > li:last-child > :last-child, .sidebarblock > .content .ulist > ul > li:last-child > :last-child, .sidebarblock > .content .qlist > ol > li:last-child > :last-child { margin-bottom: 0; }

.literalblock pre, .listingblock pre:not(.highlight), .listingblock pre[class="highlight"], .listingblock pre[class^="highlight "], .listingblock pre.CodeRay, .listingblock pre.prettyprint { background: rgba(16, 195, 196, 0.05); }
.sidebarblock .literalblock pre, .sidebarblock .listingblock pre:not(.highlight), .sidebarblock .listingblock pre[class="highlight"], .sidebarblock .listingblock pre[class^="highlight "], .sidebarblock .listingblock pre.CodeRay, .sidebarblock .listingblock pre.prettyprint { background: #f2f1f1; }

.literalblock pre, .literalblock pre[class], .listingblock pre, .listingblock pre[class] { border: 1px solid rgba(16, 195, 196, 0.125); -webkit-border-radius: 6px; border-radius: 6px; word-wrap: break-word; padding: 1em; font-size: 0.8125em; }
.literalblock pre.nowrap, .literalblock pre[class].nowrap, .listingblock pre.nowrap, .listingblock pre[class].nowrap { overflow-x: auto; white-space: pre; word-wrap: normal; }
@media only screen and (min-width: 768px) { .literalblock pre, .literalblock pre[class], .listingblock pre, .listingblock pre[class] { font-size: 0.90625em; } }
@media only screen and (min-width: 1280px) { .literalblock pre, .literalblock pre[class], .listingblock pre, .listingblock pre[class] { font-size: 1em; } }

.literalblock.output pre { color: rgba(16, 195, 196, 0.05); background-color: inherit; }

.listingblock pre.highlightjs { padding: 0; }
.listingblock pre.highlightjs > code { padding: 1em; -webkit-border-radius: 6px; border-radius: 6px; }

.listingblock > .content { position: relative; }

.listingblock code[data-lang]:before { display: none; content: attr(data-lang); position: absolute; font-size: 0.75em; top: 0.425rem; right: 0.5rem; line-height: 1; text-transform: uppercase; color: #999; }

.listingblock:hover code[data-lang]:before { display: block; }

.listingblock.terminal pre .command:before { content: attr(data-prompt); padding-right: 0.5em; color: #999; }

.listingblock.terminal pre .command:not([data-prompt]):before { content: "$"; }

table.pyhltable { border-collapse: separate; border: 0; margin-bottom: 0; background: none; }

table.pyhltable td { vertical-align: top; padding-top: 0; padding-bottom: 0; line-height: 1.4; }

table.pyhltable td.code { padding-left: .75em; padding-right: 0; }

pre.pygments .lineno, table.pyhltable td:not(.code) { color: #999; padding-left: 0; padding-right: .5em; border-right: 1px solid rgba(145, 135, 84, 0.15); }

pre.pygments .lineno { display: inline-block; margin-right: .25em; }

table.pyhltable .linenodiv { background: none !important; padding-right: 0 !important; }

.quoteblock { margin: 0 1em 1.25em 1.5em; display: table; }
.quoteblock > .title { margin-left: -1.5em; margin-bottom: 0.75em; }
.quoteblock blockquote, .quoteblock blockquote p { color: #999999; font-size: 1.15rem; line-height: 1.75; word-spacing: 0.1em; letter-spacing: 0; font-style: italic; text-align: justify; }
.quoteblock blockquote { margin: 0; padding: 0; border: 0; }
.quoteblock blockquote:before { content: "\201c"; float: left; font-size: 2.75em; font-weight: bold; line-height: 0.6em; margin-left: -0.6em; color: #002c5e; text-shadow: 0 1px 2px rgba(0, 0, 0, 0.1); }
.quoteblock blockquote > .paragraph:last-child p { margin-bottom: 0; }
.quoteblock .attribution { margin-top: 0.5em; margin-right: 0.5ex; text-align: right; }
.quoteblock .quoteblock { margin-left: 0; margin-right: 0; padding: 0.5em 0; border-left: 3px solid #666666; }
.quoteblock .quoteblock blockquote { padding: 0 0 0 0.75em; }
.quoteblock .quoteblock blockquote:before { display: none; }

.verseblock { margin: 0 1em 1.25em 1em; }
.verseblock pre { font-family: "Open Sans", "DejaVu Sans", sans; font-size: 1.15rem; color: #999999; font-weight: 300; text-rendering: optimizeLegibility; }
.verseblock pre strong { font-weight: 400; }
.verseblock .attribution { margin-top: 1.25rem; margin-left: 0.5ex; }

.quoteblock .attribution, .verseblock .attribution { font-size: 0.8125em; line-height: 1.45; font-style: italic; }
.quoteblock .attribution br, .verseblock .attribution br { display: none; }
.quoteblock .attribution cite, .verseblock .attribution cite { display: block; letter-spacing: -0.025em; color: #666666; }

.quoteblock.abstract { margin: 0 0 1.25em 0; display: block; }
.quoteblock.abstract blockquote, .quoteblock.abstract blockquote p { text-align: left; word-spacing: 0; }
.quoteblock.abstract blockquote:before, .quoteblock.abstract blockquote p:first-of-type:before { display: none; }

table.tableblock { max-width: 100%; border-collapse: separate; }
table.tableblock td > .paragraph:last-child p > p:last-child, table.tableblock th > p:last-child, table.tableblock td > p:last-child { margin-bottom: 0; }

table.tableblock, th.tableblock, td.tableblock { border: 0 solid rgba(145, 135, 84, 0.15); }

table.grid-all th.tableblock, table.grid-all td.tableblock { border-width: 0 1px 1px 0; }

table.grid-all tfoot > tr > th.tableblock, table.grid-all tfoot > tr > td.tableblock { border-width: 1px 1px 0 0; }

table.grid-cols th.tableblock, table.grid-cols td.tableblock { border-width: 0 1px 0 0; }

table.grid-all * > tr > .tableblock:last-child, table.grid-cols * > tr > .tableblock:last-child { border-right-width: 0; }

table.grid-rows th.tableblock, table.grid-rows td.tableblock { border-width: 0 0 1px 0; }

table.grid-all tbody > tr:last-child > th.tableblock, table.grid-all tbody > tr:last-child > td.tableblock, table.grid-all thead:last-child > tr > th.tableblock, table.grid-rows tbody > tr:last-child > th.tableblock, table.grid-rows tbody > tr:last-child > td.tableblock, table.grid-rows thead:last-child > tr > th.tableblock { border-bottom-width: 0; }

table.grid-rows tfoot > tr > th.tableblock, table.grid-rows tfoot > tr > td.tableblock { border-width: 1px 0 0 0; }

table.frame-all { border-width: 1px; }

table.frame-sides { border-width: 0 1px; }

table.frame-topbot { border-width: 1px 0; }

th.halign-left, td.halign-left { text-align: left; }

th.halign-right, td.halign-right { text-align: right; }

th.halign-center, td.halign-center { text-align: center; }

th.valign-top, td.valign-top { vertical-align: top; }

th.valign-bottom, td.valign-bottom { vertical-align: bottom; }

th.valign-middle, td.valign-middle { vertical-align: middle; }

table thead th, table tfoot th { font-weight: bold; }

tbody tr th { display: table-cell; line-height: 1.5; background: rgba(119, 84, 22, 0.1); }

tbody tr th, tbody tr th p, tfoot tr th, tfoot tr th p { color: rgba(0, 0, 0, 0.8); font-weight: bold; }

p.tableblock > code:only-child { background: none; padding: 0; }

p.tableblock { font-size: 1em; }

td > div.verse { white-space: pre; }

ol { margin-left: 1.75em; }

ul li ol { margin-left: 1.5em; }

dl dd { margin-left: 1.125em; }

dl dd:last-child, dl dd:last-child > :last-child { margin-bottom: 0; }

ol > li p, ul > li p, ul dd, ol dd, .olist .olist, .ulist .ulist, .ulist .olist, .olist .ulist { margin-bottom: 0.625em; }

ul.unstyled, ol.unnumbered, ul.checklist, ul.none { list-style-type: none; }

ul.unstyled, ol.unnumbered, ul.checklist { margin-left: 0.625em; }

ul.checklist li > p:first-child > .fa-square-o:first-child, ul.checklist li > p:first-child > .fa-check-square-o:first-child { width: 1em; font-size: 0.85em; }

ul.checklist li > p:first-child > input[type="checkbox"]:first-child { width: 1em; position: relative; top: 1px; }

ul.inline { margin: 0 auto 0.625em auto; margin-left: -1.375em; margin-right: 0; padding: 0; list-style: none; overflow: hidden; }
ul.inline > li { list-style: none; float: left; margin-left: 1.375em; display: block; }
ul.inline > li > * { display: block; }

.unstyled dl dt { font-weight: normal; font-style: normal; }

ol.arabic { list-style-type: decimal; }

ol.decimal { list-style-type: decimal-leading-zero; }

ol.loweralpha { list-style-type: lower-alpha; }

ol.upperalpha { list-style-type: upper-alpha; }

ol.lowerroman { list-style-type: lower-roman; }

ol.upperroman { list-style-type: upper-roman; }

ol.lowergreek { list-style-type: lower-greek; }

.hdlist > table, .colist > table { border: 0; background: none; }
.hdlist > table > tbody > tr, .colist > table > tbody > tr { background: none; }

td.hdlist1, td.hdlist2 { vertical-align: top; padding: 0 0.625em; }

td.hdlist1 { font-weight: bold; padding-bottom: 1.25em; }

.literalblock + .colist, .listingblock + .colist { margin-top: -0.5em; }

.colist > table tr > td:first-of-type { padding: 0 0.75em; line-height: 1; }
.colist > table tr > td:last-of-type { padding: 0.25em 0; }

.thumb, .th { line-height: 0; display: inline-block; border: solid 4px white; -webkit-box-shadow: 0 0 0 1px #dddddd; box-shadow: 0 0 0 1px #dddddd; }

.imageblock.left, .imageblock[style*="float: left"] { margin: 0.25em 0.625em 1.25em 0; }
.imageblock.right, .imageblock[style*="float: right"] { margin: 0.25em 0 1.25em 0.625em; }
.imageblock > .title { margin-bottom: 0; }
.imageblock.thumb, .imageblock.th { border-width: 6px; }
.imageblock.thumb > .title, .imageblock.th > .title { padding: 0 0.125em; }

.image.left, .image.right { margin-top: 0.25em; margin-bottom: 0.25em; display: inline-block; line-height: 0; }
.image.left { margin-right: 0.625em; }
.image.right { margin-left: 0.625em; }

a.image { text-decoration: none; display: inline-block; }
a.image object { pointer-events: none; }

sup.footnote, sup.footnoteref { font-size: 0.875em; position: static; vertical-align: super; }
sup.footnote a, sup.footnoteref a { text-decoration: none; }
sup.footnote a:active, sup.footnoteref a:active { text-decoration: underline; }

#footnotes { padding-top: 0.75em; padding-bottom: 0.75em; margin-bottom: 0.625em; }
#footnotes hr { width: 20%; min-width: 6.25em; margin: -0.25em 0 0.75em 0; border-width: 1px 0 0 0; }
#footnotes .footnote { padding: 0 0.375em 0 0.225em; line-height: 1.3334; font-size: 0.875em; margin-left: 1.2em; text-indent: -1.05em; margin-bottom: 0.2em; }
#footnotes .footnote a:first-of-type { font-weight: bold; text-decoration: none; }
#footnotes .footnote:last-of-type { margin-bottom: 0; }

#content #footnotes { margin-top: -0.625em; margin-bottom: 0; padding: 0.75em 0; }

.gist .file-data > table { border: 0; background: #fff; width: 100%; margin-bottom: 0; }
.gist .file-data > table td.line-data { width: 99%; }

div.unbreakable { page-break-inside: avoid; }

.big { font-size: larger; }

.small { font-size: smaller; }

.underline { text-decoration: underline; }

.overline { text-decoration: overline; }

.line-through { text-decoration: line-through; }

.aqua { color: #00bfbf; }

.aqua-background { background-color: #00fafa; }

.black { color: black; }

.black-background { background-color: black; }

.blue { color: #0000bf; }

.blue-background { background-color: #0000fa; }

.fuchsia { color: #bf00bf; }

.fuchsia-background { background-color: #fa00fa; }

.gray { color: #606060; }

.gray-background { background-color: #7d7d7d; }

.green { color: #006000; }

.green-background { background-color: #007d00; }

.lime { color: #00bf00; }

.lime-background { background-color: #00fa00; }

.maroon { color: #600000; }

.maroon-background { background-color: #7d0000; }

.navy { color: #000060; }

.navy-background { background-color: #00007d; }

.olive { color: #606000; }

.olive-background { background-color: #7d7d00; }

.purple { color: #600060; }

.purple-background { background-color: #7d007d; }

.red { color: #bf0000; }

.red-background { background-color: #fa0000; }

.silver { color: #909090; }

.silver-background { background-color: #bcbcbc; }

.teal { color: #006060; }

.teal-background { background-color: #007d7d; }

.white { color: #bfbfbf; }

.white-background { background-color: #fafafa; }

.yellow { color: #bfbf00; }

.yellow-background { background-color: #fafa00; }

span.icon > .fa { cursor: default; }

.admonitionblock td.icon [class^="fa icon-"] { font-size: 2.5em; text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.5); cursor: default; }
.admonitionblock td.icon .icon-note:before { content: "\f05a"; color: #004060; }
.admonitionblock td.icon .icon-tip:before { content: "\f0eb"; text-shadow: 1px 1px 2px rgba(155, 155, 0, 0.8); color: #111; }
.admonitionblock td.icon .icon-warning:before { content: "\f071"; color: #bf6900; }
.admonitionblock td.icon .icon-caution:before { content: "\f06d"; color: #bf3400; }
.admonitionblock td.icon .icon-important:before { content: "\f06a"; color: #bf0000; }

.conum[data-value] { display: inline-block; color: #fff !important; background-color: rgba(0, 0, 0, 0.8); -webkit-border-radius: 100px; border-radius: 100px; text-align: center; font-size: 0.75em; width: 1.67em; height: 1.67em; line-height: 1.67em; font-family: "Open Sans", "DejaVu Sans", sans-serif; font-style: normal; font-weight: bold; }
.conum[data-value] * { color: #fff !important; }
.conum[data-value] + b { display: none; }
.conum[data-value]:after { content: attr(data-value); }
pre .conum[data-value] { position: relative; top: -0.125em; }

b.conum * { color: inherit !important; }

.conum:not([data-value]):empty { display: none; }

#toc.toc2 ul ul { list-style-type: circle; padding-left: 2em; }

.sect1 { padding-bottom: 0 !important; margin-bottom: 2.5em; }

#header h1 { font-weight: bold; position: relative; left: -0.0625em; }

#content h2, #content h3, #content #toctitle, #content .sidebarblock > .content > .title, #content h4, #content h5, #content #toctitle { position: relative; left: -0.0625em; }
#content h2 { font-weight: bold; }

.paragraph.lead > p, #preamble > .sectionbody > .paragraph:first-of-type p { color: black; }

pre.pygments.highlight { background-color: rgba(16, 195, 196, 0.05); }

.pygments .tok-err { border: none !important; color: #800000 !important; }

.pygments .tok-c { color: #999 !important; }

</style>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.4.0/css/font-awesome.min.css">
<style>
/* Stylesheet for CodeRay to match GitHub theme | MIT License | http://foundation.zurb.com */
/*pre.CodeRay {background-color:#f7f7f8;}*/
.CodeRay .line-numbers{border-right:1px solid #d8d8d8;padding:0 0.5em 0 .25em}
.CodeRay span.line-numbers{display:inline-block;margin-right:.5em;color:rgba(0,0,0,.3)}
.CodeRay .line-numbers strong{color:rgba(0,0,0,.4)}
table.CodeRay{border-collapse:separate;border-spacing:0;margin-bottom:0;border:0;background:none}
table.CodeRay td{vertical-align: top;line-height:1.45}
table.CodeRay td.line-numbers{text-align:right}
table.CodeRay td.line-numbers>pre{padding:0;color:rgba(0,0,0,.3)}
table.CodeRay td.code{padding:0 0 0 .5em}
table.CodeRay td.code>pre{padding:0}
.CodeRay .debug{color:#fff !important;background:#000080 !important}
.CodeRay .annotation{color:#007}
.CodeRay .attribute-name{color:#000080}
.CodeRay .attribute-value{color:#700}
.CodeRay .binary{color:#509}
.CodeRay .comment{color:#998;font-style:italic}
.CodeRay .char{color:#04d}
.CodeRay .char .content{color:#04d}
.CodeRay .char .delimiter{color:#039}
.CodeRay .class{color:#458;font-weight:bold}
.CodeRay .complex{color:#a08}
.CodeRay .constant,.CodeRay .predefined-constant{color:#008080}
.CodeRay .color{color:#099}
.CodeRay .class-variable{color:#369}
.CodeRay .decorator{color:#b0b}
.CodeRay .definition{color:#099}
.CodeRay .delimiter{color:#000}
.CodeRay .doc{color:#970}
.CodeRay .doctype{color:#34b}
.CodeRay .doc-string{color:#d42}
.CodeRay .escape{color:#666}
.CodeRay .entity{color:#800}
.CodeRay .error{color:#808}
.CodeRay .exception{color:inherit}
.CodeRay .filename{color:#099}
.CodeRay .function{color:#900;font-weight:bold}
.CodeRay .global-variable{color:#008080}
.CodeRay .hex{color:#058}
.CodeRay .integer,.CodeRay .float{color:#099}
.CodeRay .include{color:#555}
.CodeRay .inline{color:#000}
.CodeRay .inline .inline{background:#ccc}
.CodeRay .inline .inline .inline{background:#bbb}
.CodeRay .inline .inline-delimiter{color:#d14}
.CodeRay .inline-delimiter{color:#d14}
.CodeRay .important{color:#555;font-weight:bold}
.CodeRay .interpreted{color:#b2b}
.CodeRay .instance-variable{color:#008080}
.CodeRay .label{color:#970}
.CodeRay .local-variable{color:#963}
.CodeRay .octal{color:#40e}
.CodeRay .predefined{color:#369}
.CodeRay .preprocessor{color:#579}
.CodeRay .pseudo-class{color:#555}
.CodeRay .directive{font-weight:bold}
.CodeRay .type{font-weight:bold}
.CodeRay .predefined-type{color:inherit}
.CodeRay .reserved,.CodeRay .keyword {color:#000;font-weight:bold}
.CodeRay .key{color:#808}
.CodeRay .key .delimiter{color:#606}
.CodeRay .key .char{color:#80f}
.CodeRay .value{color:#088}
.CodeRay .regexp .delimiter{color:#808}
.CodeRay .regexp .content{color:#808}
.CodeRay .regexp .modifier{color:#808}
.CodeRay .regexp .char{color:#d14}
.CodeRay .regexp .function{color:#404;font-weight:bold}
.CodeRay .string{color:#d20}
.CodeRay .string .string .string{background:#ffd0d0}
.CodeRay .string .content{color:#d14}
.CodeRay .string .char{color:#d14}
.CodeRay .string .delimiter{color:#d14}
.CodeRay .shell{color:#d14}
.CodeRay .shell .delimiter{color:#d14}
.CodeRay .symbol{color:#990073}
.CodeRay .symbol .content{color:#a60}
.CodeRay .symbol .delimiter{color:#630}
.CodeRay .tag{color:#008080}
.CodeRay .tag-special{color:#d70}
.CodeRay .variable{color:#036}
.CodeRay .insert{background:#afa}
.CodeRay .delete{background:#faa}
.CodeRay .change{color:#aaf;background:#007}
.CodeRay .head{color:#f8f;background:#505}
.CodeRay .insert .insert{color:#080}
.CodeRay .delete .delete{color:#800}
.CodeRay .change .change{color:#66f}
.CodeRay .head .head{color:#f4f}
</style>
</head>
<body class="book toc2 toc-right">
<div id="header">
<h1>vmx tutorial</h1>
<div class="details">
<span id="author" class="author">Ping Song</span><br>
<span id="email" class="email"><a href="mailto:pings@juniper.net">pings@juniper.net</a></span><br>
</div>
<div id="toc" class="toc2">
<div id="toctitle">Table of Content</div>
<ul class="sectlevel1">
<li><a href="#_about_this_doc">about This doc :</a></li>
<li><a href="#_part_1_vmx_installation">Part 1: VMX installation</a>
<ul class="sectlevel1">
<li><a href="#preparation">1. prepare for the installation</a>
<ul class="sectlevel2">
<li><a href="#_identify_current_system_info">1.1. identify current system info</a>
<ul class="sectlevel3">
<li><a href="#_server_manufacturer_model_sn">1.1.1. server Manufacturer/model/SN</a></li>
<li><a href="#_operating_system">1.1.2. Operating System</a></li>
<li><a href="#_cpu_and_memory">1.1.3. cpu and memory</a></li>
<li><a href="#_network_adapter_controller">1.1.4. network adapter/controller</a></li>
<li><a href="#_ixgbe_kernel_driver">1.1.5. ixgbe kernel driver</a></li>
<li><a href="#IOMMU">1.1.6. VT-d/IOMMU</a></li>
<li><a href="#_kvm_qemu_software_version">1.1.7. kvm/qemu software version</a></li>
<li><a href="#_libvirt">1.1.8. libvirt</a></li>
</ul>
</li>
<li><a href="#_adjust_the_system">1.2. adjust the system</a>
<ul class="sectlevel3">
<li><a href="#_bios_setting">1.2.1. BIOS setting</a></li>
<li><a href="#enable-iommu">1.2.2. enable iommu/VT-d</a></li>
<li><a href="#install-linux-kernel">1.2.3. install required linux kernel for SR-IOV based VMX</a></li>
<li><a href="#_install_required_software_packages">1.2.4. install required software packages</a></li>
<li><a href="#_libvirt_2">1.2.5. libvirt</a></li>
</ul>
</li>
<li><a href="#_download_vmx_installation_package">1.3. download vmx installation package</a></li>
<li><a href="#_prepare_a_work_folder_for_the_installation">1.4. prepare a "work folder" for the installation</a></li>
</ul>
</li>
<li><a href="#_install_vmx_using_installation_script_sr_iov">2. install VMX using installation script (SR-IOV)</a>
<ul class="sectlevel2">
<li><a href="#_the_vmx_conf_file">2.1. the vmx.conf file</a></li>
<li><a href="#_assigning_mac_address">2.2. assigning MAC address</a></li>
<li><a href="#_modify_the_vmx_conf_sr_iov">2.3. modify the vmx.conf (SR-IOV)</a></li>
<li><a href="#_run_the_installation_script_sr_iov">2.4. run the installation script: SR-IOV</a></li>
<li><a href="#_quick_verification">2.5. quick verification</a></li>
<li><a href="#_uninstall_cleanup_vmx_with_installation_script">2.6. uninstall (cleanup) VMX with installation script</a></li>
</ul>
</li>
<li><a href="#_install_vmx_using_installation_script_virtio">3. install VMX using installation script (VIRTIO)</a>
<ul class="sectlevel2">
<li><a href="#_modify_the_vmx_conf_virtio">3.1. modify the vmx.conf (virtio)</a></li>
<li><a href="#_run_the_installation_script_virtio">3.2. run the installation script: virtio</a></li>
<li><a href="#_uninstall_cleanup_vmx_with_installation_script_2">3.3. uninstall (cleanup) VMX with installation script</a></li>
</ul>
</li>
<li><a href="#_setup_multiple_vmx_instances">4. setup multiple VMX instances</a>
<ul class="sectlevel2">
<li><a href="#_config_file_for_the_first_vmx_instance">4.1. config file for the first VMX instance</a></li>
<li><a href="#_config_file_for_the_second_vmx_instance">4.2. config file for the second VMX instance</a></li>
<li><a href="#_run_installation_script_with_code_cfg_code_option">4.3. run installation script with <code>--cfg</code> option</a></li>
<li><a href="#_verify_the_2_running_vmx">4.4. verify the 2 running VMX</a>
<ul class="sectlevel3">
<li><a href="#_ifconfig_virtio">4.4.1. ifconfig (virtio)</a></li>
<li><a href="#_virtio_bridging_linux_bridge">4.4.2. virtio bridging: linux bridge</a></li>
<li><a href="#routing-test">4.4.3. routing test</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#install-VMX-manually">5. install VMX manually</a>
<ul class="sectlevel2">
<li><a href="#_script_generated_xml_files">5.1. script generated XML files</a></li>
<li><a href="#_manual_installation_steps">5.2. manual installation steps</a></li>
<li><a href="#_verify_installations">5.3. verify installations</a></li>
<li><a href="#_internal_and_external_bridges_in_multiple_tenant_environment">5.4. internal and external bridges in multiple tenant environment</a></li>
<li><a href="#_manually_uninstall_vmxs">5.5. manually uninstall VMXs</a></li>
</ul>
</li>
<li><a href="#_upgrading_vmx">6. upgrading VMX</a></li>
<li><a href="#troubleshooting-install">7. troubleshooting installation script</a>
<ul class="sectlevel2">
<li><a href="#_ixgbe_compilation_error">7.1. ixgbe compilation error</a>
<ul class="sectlevel3">
<li><a href="#_analysis_and_solution">7.1.1. analysis and solution</a></li>
</ul>
</li>
<li><a href="#_82599_nic_not_recognized">7.2. 82599 NIC not recognized</a>
<ul class="sectlevel3">
<li><a href="#_analysis_and_solution_2">7.2.1. analysis and solution</a></li>
</ul>
</li>
<li><a href="#_hyperthread">7.3. hyperthread</a>
<ul class="sectlevel3">
<li><a href="#_analysis_and_solution_3">7.3.1. analysis and solution:</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#_part_2_vmx_verification">Part 2: VMX verification</a>
<ul class="sectlevel1">
<li><a href="#_vcp_vfp_guest_vm_overview">8. VCP/VFP guest VM overview</a></li>
<li><a href="#_login_to_the_vnc_console">9. Login to the VNC console</a></li>
<li><a href="#_ixgbe_driver_and_ixgbe_driven_interfaces">10. ixgbe driver and ixgbe-driven interfaces</a>
<ul class="sectlevel2">
<li><a href="#_legacy_code_ifconfig_code_command">10.1. legacy <code>ifconfig</code> command</a></li>
<li><a href="#__code_ip_code_tool_sr_iov">10.2. <code>ip</code> tool (SR-IOV)</a></li>
<li><a href="#_host_interfaces_virtio">10.3. host interfaces (virtio)</a></li>
<li><a href="#_vmx_fxp0_and_em1_interface">10.4. VMX fxp0 and em1 interface</a></li>
<li><a href="#__code_ge_x_y_z_code_interface">10.5. <code>ge-x/y/z</code> interface</a></li>
</ul>
</li>
<li><a href="#_virtual_networks_bridging_sr_iov">11. virtual networks/bridging (SR-IOV)</a>
<ul class="sectlevel2">
<li><a href="#_mapping_of_bridge_tap_guest_vnic_sr_iov">11.1. mapping of bridge, tap, guest vNIC (SR-IOV)</a></li>
</ul>
</li>
<li><a href="#VCPU_ESSENTIAL">12. vcpu essential</a></li>
<li><a href="#_memory">13. memory</a></li>
<li><a href="#_virtio_bridging">14. virtio bridging</a>
<ul class="sectlevel2">
<li><a href="#_build_a_setup_with_internal_connections">14.1. build a setup with internal connections</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#_part_3_vmx_kvm_features">Part 3: VMX/KVM features</a>
<ul class="sectlevel1">
<li><a href="#_vm_management_virsh">15. VM management - virsh</a>
<ul class="sectlevel2">
<li><a href="#_virsh_basic_concept">15.1. virsh basic concept</a></li>
<li><a href="#_libvirt_virsh_commonly_used_commands">15.2. libvirt/virsh commonly used commands</a>
<ul class="sectlevel3">
<li><a href="#_domain_managment">15.2.1. domain managment</a></li>
<li><a href="#_node_management">15.2.2. node management</a></li>
<li><a href="#_network_and_interface_management">15.2.3. network and interface management</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#VT-D">16. iommu/VT-d</a>
<ul class="sectlevel2">
<li><a href="#_vt_d_basic_concept">16.1. VT-d basic concept</a>
<ul class="sectlevel3">
<li><a href="#_software_based_approach">16.1.1. software based approach</a></li>
<li><a href="#_direct_assignment">16.1.2. direct assignment</a></li>
</ul>
</li>
<li><a href="#_pci_stub">16.2. pci_stub</a></li>
<li><a href="#_process_to_assign_pci_device_in_kvm">16.3. process to assign PCI device in KVM</a></li>
</ul>
</li>
<li><a href="#_sriov">17. SRIOV</a>
<ul class="sectlevel2">
<li><a href="#_sriov_basic_concept">17.1. SRIOV basic concept</a>
<ul class="sectlevel3">
<li><a href="#_what_is_io_virtualization">17.1.1. what is "IO virtualization"?</a></li>
<li><a href="#_what_is_a_function">17.1.2. what is a "function"?</a></li>
<li><a href="#_pci_functions_defined_in_sr_iov">17.1.3. PCI functions defined in SR-IOV</a></li>
<li><a href="#_what_is_a_root">17.1.4. what is a "root"?</a></li>
<li><a href="#_why_is_it_called_single">17.1.5. why is it called "single"?</a></li>
<li><a href="#_corelation_with_code_vt_d_code">17.1.6. corelation with <code>VT-d</code></a></li>
<li><a href="#_other_requirement">17.1.7. other requirement</a></li>
</ul>
</li>
<li><a href="#_sr_iov_packet_flow">17.2. SR-IOV packet flow</a></li>
<li><a href="#_sr_iov_configuration_in_vmx">17.3. SR-IOV configuration in VMX</a></li>
<li><a href="#_example_of_1_vf">17.4. example of 1 VF</a></li>
<li><a href="#_example_of_4_vfs">17.5. example of 4 VFs</a>
<ul class="sectlevel3">
<li><a href="#_the_rule_of_thumb_for_4_vf">17.5.1. the "rule of thumb" for 4 VF</a></li>
</ul>
</li>
<li><a href="#_example_of_8_vfs">17.6. example of 8 VFs:</a>
<ul class="sectlevel3">
<li><a href="#_the_rule_of_thumb_for_8_vf">17.6.1. the "rule of thumb" for 8 VF</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#VIRTIO">18. virtio</a>
<ul class="sectlevel2">
<li><a href="#_virtio_basic_concept">18.1. virtio basic concept</a></li>
<li><a href="#_virtio_verification_in_vpfe_guest_vm">18.2. virtio verification in vPFE guest VM</a></li>
</ul>
</li>
<li><a href="#_cpu_pinning_affinitization">19. cpu pinning (affinitization)</a>
<ul class="sectlevel2">
<li><a href="#_2_terms">19.1. 2 terms</a></li>
<li><a href="#_2_code_virsh_code_commands">19.2. 2 <code>virsh</code> commands</a></li>
<li><a href="#_a_simple_test">19.3. a simple test</a></li>
</ul>
</li>
<li><a href="#_hugepage">20. hugepage</a>
<ul class="sectlevel2">
<li><a href="#_hugepage_basic_concept">20.1. hugepage basic concept</a></li>
<li><a href="#_hugepage_allocation">20.2. hugepage allocation</a></li>
<li><a href="#_change_hugepages_number">20.3. change hugepages number</a></li>
<li><a href="#_hugepage_and_numa">20.4. hugepage and numa</a></li>
<li><a href="#_a_simple_test_of_hugepage">20.5. a simple test of hugepage</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#appendix">appendix</a>
<ul class="sectlevel1">
<li><a href="#_reference">reference</a></li>
<li><a href="#folder-structure">VMX package folder structure</a></li>
<li><a href="#_vmx_installation_script_generated_xml_file">VMX installation script generated XML file</a>
<ul class="sectlevel2">
<li><a href="#xml-files">generated vRE xml</a></li>
<li><a href="#_generated_vpfe_xml">generated vPFE xml</a></li>
<li><a href="#_generated_virtual_network_xml">generated virtual network XML</a></li>
<li><a href="#shell-files">generated shell scripts</a></li>
<li><a href="#_generated_files_for_virtio">generated files for VIRTIO</a></li>
<li><a href="#_dumpxml_vcp_vmx1">dumpxml vcp-vmx1</a></li>
<li><a href="#_dumpxml_vfp_vmx1">dumpxml vfp-vmx1</a></li>
</ul>
</li>
<li><a href="#__code_virsh_capabilities_code_complete_output"><code>virsh capabilities</code> complete output</a></li>
<li><a href="#_ixgbe_driver_issue">ixgbe driver issue</a></li>
<li><a href="#_end">end</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<table class="tableblock frame-all grid-all spread">
<colgroup>
<col style="width: 25%;">
<col style="width: 25%;">
<col style="width: 25%;">
<col style="width: 25%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">version</th>
<th class="tableblock halign-left valign-top">author</th>
<th class="tableblock halign-left valign-top">date</th>
<th class="tableblock halign-left valign-top">changelog</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Version 0.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="mailto:pings@juniper.net">pings@juniper.net</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">(2015-11-23)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">main part of the doc done</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Version 0.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="mailto:pings@juniper.net">pings@juniper.net</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">(2015-11-28)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">add "references"</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Version 0.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="mailto:pings@juniper.net">pings@juniper.net</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">(2016-01-04)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">add qemu-kvm params breakdown(TODO)</p></td>
</tr>
</tbody>
</table>
<div style="page-break-after: always;"></div>
<div class="paragraph">
<p>Kernel-based Virtual Machine (KVM) is a virtualization infrastructure for the
Linux kernel that turns it into a hypervisor. It has been gaining industry
traction and market share in a wide variety of software deployments in just a
few short years. KVM turns to be a very interesting virtualization topic and is
well positioned for the future.</p>
</div>
<div class="paragraph">
<p>Juniper MX86(VMX: Virtual MX) is a virtual version of the MX Series 3D
Universal Edge Router that is installed on an industry-standard x86 server
running a Linux operating system, applicable third-party software, and the KVM
hypervisor. It is one of industry-level implementations of router based on KVM
technology.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_about_this_doc"><a class="anchor" href="#_about_this_doc"></a>about This doc :</h2>
<div class="sectionbody">
<div class="paragraph">
<p>This doc records some learning and testing notes about KVM/MX86 technology.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>illustration of various installation procedures</p>
</li>
<li>
<p>illustration of verification procedure</p>
</li>
<li>
<p>illustration of features involved in VMX installation process, mostly from
KVM/virtualization technology&#8217;s perspecive</p>
</li>
<li>
<p>some case studies</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This doc can be used as reference when you:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>read the official VMX documents, use this as an "accompanying" doc</p>
</li>
<li>
<p>need a quick list of commands to verify a feature or troubleshoot an issue</p>
</li>
<li>
<p>when run into issues, need a "working example" as a reference</p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">a "TODO" list:</div>
<ul>
<li>
<p>breakdown of qemu command parameters</p>
</li>
<li>
<p>customerization of the installation script to make it startup friendly</p>
</li>
<li>
<p>libvirt API scripting</p>
</li>
<li>
<p>DPDK programming</p>
</li>
<li>
<p>internal crabs? (flow cache, junos troubleshooting, internal packet flow, etc)</p>
</li>
</ul>
</div>
<div style="page-break-after: always;"></div>
</div>
</div>
<h1 id="_part_1_vmx_installation" class="sect0"><a class="anchor" href="#_part_1_vmx_installation"></a>Part 1: VMX installation</h1>
<div class="sect1">
<h2 id="preparation"><a class="anchor" href="#preparation"></a>1. prepare for the installation</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_identify_current_system_info"><a class="anchor" href="#_identify_current_system_info"></a>1.1. identify current system info</h3>
<div class="paragraph">
<p>before starting the installation, we need to collect some basic information
about the server that VMX is about to be built on. At mininum, make sure the
server complies to the <a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/reference/general/vmx-hw-sw-minimums.html">"Minimum Hardware and Software
Requirements"</a>.</p>
</div>
<div class="ulist checklist">
<div class="title">pre-installation checklist</div>
<ul class="checklist">
<li>
<p><i class="fa fa-square-o"></i> server manufacturer info</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> operating system</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> cpu and memory</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> NIC/controller</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> NIC driver</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> VT-d/IOMMU</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> kvm/qemu version</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> libvirt version</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="_server_manufacturer_model_sn"><a class="anchor" href="#_server_manufacturer_model_sn"></a>1.1.1. server Manufacturer/model/SN</h4>
<div class="ulist">
<div class="title">this server&#8217;s manufacturer/model info:</div>
<ul>
<li>
<p>this server is a <code>ProLiant BL660c Gen8</code> server.</p>
</li>
<li>
<p>chassis SN is <code>USE4379WSS</code></p>
</li>
</ul>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
same info can be acquired from HP <code>ILO</code>, but command from ssh session is
more convenient.
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>    ping@trinity:~$ sudo dmidecode |sed -n '/System Info/,/^$/p'
<span class="line-numbers"> 2</span>    System Information
<span class="line-numbers"> 3</span>            Manufacturer: HP
<span class="line-numbers"> 4</span>            Product Name: ProLiant BL660c Gen8          #&lt;------
<span class="line-numbers"> 5</span>            Version: Not Specified
<span class="line-numbers"> 6</span>            Serial Number: USE4379WSS                   #&lt;------
<span class="line-numbers"> 7</span>            UUID: 31393736-3831-5355-4534-333739575353
<span class="line-numbers"> 8</span>            Wake-up Type: Power Switch
<span class="line-numbers"> 9</span>            SKU Number: 679118-B21
<span class="line-numbers">10</span>            Family: ProLiant</code></pre>
</div>
</div>
<div class="paragraph">
<p><code>dmidecode -t 1</code> will print same info:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>    ping@trinity:~$ sudo dmidecode -t 1
<span class="line-numbers"> 2</span>        Handle 0x0100, DMI type 1, 27 bytes
<span class="line-numbers"> 3</span>        System Information
<span class="line-numbers"> 4</span>                Manufacturer: HP
<span class="line-numbers"> 5</span>                Product Name: ProLiant BL660c Gen8
<span class="line-numbers"> 6</span>                Version: Not Specified
<span class="line-numbers"> 7</span>                Serial Number: USE4379WSS
<span class="line-numbers"> 8</span>                UUID: 31393736-3831-5355-4534-333739575353
<span class="line-numbers"> 9</span>                Wake-up Type: Power Switch
<span class="line-numbers">10</span>                SKU Number: 679118-B21
<span class="line-numbers">11</span>                Family: ProLiant</code></pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<code>-t</code> is handy , but the use of <code>sed</code> is a more general way to extract a
part of text from long text output of any command.
</td>
</tr>
</table>
</div>
<div class="sidebarblock">
<div class="content">
<div class="title">dmidecode tool</div>
<div class="paragraph">
<p>most hardware data can be queried by <code>dmidecode</code> and/or <code>lshw</code> command.  the
<code>dmi type</code> code of <code>system info</code> is 1, so <code>dmidecode -t 1</code> will print same
info. refer to manpage of <code>dmidecode</code> for more info about the usage.</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The SMBIOS specification defines the following DMI types:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Type   Inf         0   BIOS
   1   System
   2   Baseboard
   3   Chassis
   4   Processor
   5   Memory Controller
   6   Memory Module
   7   Cache
   8   Port Connector
   9   System Slots
  10   On Board Devices
  11   OEM Strings
  12   System Configuration Options
  13   BIOS Language
  14   Group Associations
  16   System Event Log
  16   Physical Memory Array
  17   Memory Device
  18   32-bit Memory Error
  19   Memory Array Mapped Address
  20   Memory Device Mapped Address
  21   Built-in Pointing Device
  22   Portable Battery
  23   System Reset
  24   Hardware Security
  25   System Power Controls
  26   Voltage Probe
  27   Cooling Device
  28   Temperature Probe
  29   Electrical Current Probe
  30   Out-of-band Remote Access
  31   Boot Integrity Services
  32   System Boot
  33   64-bit Memory Error
  34   Management Device
  35   Management Device Component
  36   Management Device Threshold Data
  37   Memory Channel
  38   IPMI Device
  39   Power Supply
  40   Additional Information
  41   Onboard Devices Extended Information
  42   Management Controller Host Interface</pre>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_operating_system"><a class="anchor" href="#_operating_system"></a>1.1.2. Operating System</h4>
<div class="ulist">
<div class="title">this server&#8217;s current operation system info:</div>
<ul>
<li>
<p>linux distribution: ubuntu 14.04.2</p>
</li>
<li>
<p>linux kernel in use: <code>3.19.0-25-generic</code></p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ lsb_release -a
<span class="line-numbers"> 2</span>No LSB modules are available.
<span class="line-numbers"> 3</span>Distributor ID: Ubuntu
<span class="line-numbers"> 4</span>Description:    Ubuntu 14.04.2 LTS  #&lt;------
<span class="line-numbers"> 5</span>Release:        14.04
<span class="line-numbers"> 6</span>Codename:       trusty
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>ping@trinity:~$ uname -a
<span class="line-numbers"> 9</span>Linux trinity 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
<span class="line-numbers">10</span>              ^^^^^^^^^</code></pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>there are many other alternative commands to print OS version/release info:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>    ping@ubuntu1:~$ cat /etc/os-release
<span class="line-numbers"> 2</span>    NAME=&quot;Ubuntu&quot;
<span class="line-numbers"> 3</span>    VERSION=&quot;14.04.1 LTS, Trusty Tahr&quot;
<span class="line-numbers"> 4</span>    ID=ubuntu
<span class="line-numbers"> 5</span>    ID_LIKE=debian
<span class="line-numbers"> 6</span>    PRETTY_NAME=&quot;Ubuntu 14.04.1 LTS&quot;
<span class="line-numbers"> 7</span>    VERSION_ID=&quot;14.04&quot;
<span class="line-numbers"> 8</span>    HOME_URL=&quot;http://www.ubuntu.com/&quot;
<span class="line-numbers"> 9</span>    SUPPORT_URL=&quot;http://help.ubuntu.com/&quot;
<span class="line-numbers">10</span>    BUG_REPORT_URL=&quot;http://bugs.launchpad.net/ubuntu/&quot;
<span class="line-numbers">11</span>
<span class="line-numbers">12</span>    ping@trinity:~$ cat /etc/lsb-release
<span class="line-numbers">13</span>    DISTRIB_ID=Ubuntu
<span class="line-numbers">14</span>    DISTRIB_RELEASE=14.04
<span class="line-numbers">15</span>    DISTRIB_CODENAME=trusty
<span class="line-numbers">16</span>    DISTRIB_DESCRIPTION=&quot;Ubuntu 14.04.2 LTS&quot;
<span class="line-numbers">17</span>
<span class="line-numbers">18</span>    ping@trinity:~$ cat /etc/issue
<span class="line-numbers">19</span>    Ubuntu 14.04.2 LTS \n \l
<span class="line-numbers">20</span>
<span class="line-numbers">21</span>    ping@trinity:~$ cat /etc/issue.net
<span class="line-numbers">22</span>    Ubuntu 14.04.2 LTS
<span class="line-numbers">23</span>
<span class="line-numbers">24</span>    ping@ubuntu1:~$ cat /proc/version
<span class="line-numbers">25</span>    Linux version 3.13.0-32-generic (buildd@kissel)
<span class="line-numbers">26</span>    (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) )
<span class="line-numbers">27</span>    #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014</code></pre>
</div>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>to list all kernels installed currently in the server:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ dpkg --get-selections | grep linux-image
<span class="line-numbers">2</span>linux-image-3.13.0-32-generic                   install     #&lt;------
<span class="line-numbers">3</span>linux-image-3.16.0-30-generic                   install     #&lt;------
<span class="line-numbers">4</span>linux-image-3.19.0-25-generic                   install     #&lt;------
<span class="line-numbers">5</span>linux-image-extra-3.13.0-32-generic             install
<span class="line-numbers">6</span>linux-image-extra-3.16.0-30-generic             install
<span class="line-numbers">7</span>linux-image-extra-3.19.0-25-generic             install
<span class="line-numbers">8</span>linux-image-generic-lts-utopic                  install</code></pre>
</div>
</div>
<div class="paragraph">
<p>In this server multiple verions of linux kernel have been installed, including
our target kernel <code>3.13.0-32-generic</code>. There is no need to install the the
kernel again - we only need to select the right one and reboot the system with
it.</p>
</div>
</div>
<div class="sect3">
<h4 id="_cpu_and_memory"><a class="anchor" href="#_cpu_and_memory"></a>1.1.3. cpu and memory</h4>
<div class="paragraph">
<p>this server&#8217;s CPU and memory info:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CPU model: Intel&#174; Xeon&#174; CPU <a href="http://ark.intel.com/products/75287">E5-4627</a>
v2 @ 3.30GHz (<a href="https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)#Server_processors">Ivy bridge</a>)</p>
</li>
<li>
<p>number of "CPU processors": 4</p>
</li>
<li>
<p>number of "physical cores" per CPU processor: 8</p>
</li>
<li>
<p>VT-x (not VT-d) enabled</p>
</li>
<li>
<p>This CPU
<a href="http://ark.intel.com/products/75287/Intel-Xeon-Processor-E5-4627-v2-16M-Cache-3_30-GHz">does
not support HT(HyperThreading)</a>, see <a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/reference/general/vmx-package-contents.html">this spec</a></p>
</li>
<li>
<p>32 32G DDR3 memory, 500+G total memory available</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ lscpu
<span class="line-numbers"> 2</span>Architecture:          x86_64
<span class="line-numbers"> 3</span>CPU op-mode(s):        32-bit, 64-bit
<span class="line-numbers"> 4</span>Byte Order:            Little Endian
<span class="line-numbers"> 5</span>CPU(s):                32           #&lt;----<i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 6</span>On-line CPU(s) list:   0-31
<span class="line-numbers"> 7</span>Thread(s) per core:    1            #&lt;----<i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 8</span>Core(s) per socket:    8
<span class="line-numbers"> 9</span>Socket(s):             4            #&lt;----<i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">10</span>NUMA node(s):          4
<span class="line-numbers">11</span>Vendor ID:             GenuineIntel
<span class="line-numbers">12</span>CPU family:            6
<span class="line-numbers">13</span>Model:                 62
<span class="line-numbers">14</span>Stepping:              4
<span class="line-numbers">15</span>CPU MHz:               3299.797
<span class="line-numbers">16</span>BogoMIPS:              6605.01
<span class="line-numbers">17</span>Virtualization:        VT-x         #&lt;----<i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">18</span>L1d cache:             32K
<span class="line-numbers">19</span>L1i cache:             32K
<span class="line-numbers">20</span>L2 cache:              256K
<span class="line-numbers">21</span>L3 cache:              16384K
<span class="line-numbers">22</span>NUMA node0 CPU(s):     0-7
<span class="line-numbers">23</span>NUMA node1 CPU(s):     8-15
<span class="line-numbers">24</span>NUMA node2 CPU(s):     16-23
<span class="line-numbers">25</span>NUMA node3 CPU(s):     24-31</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>total number of CPU cores</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>hyperthreading is not enabled or not supported in this server</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>VT-x is enabled</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>max number of sockets available (to hold CPU) in the server</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ grep -m1 &quot;model name&quot;  /proc/cpuinfo
<span class="line-numbers"> 2</span>model name      : Intel(R) Xeon(R) CPU E5-4627 v2 @ 3.30GHz   #&lt;------
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>ping@trinity:~$ cat /proc/cpuinfo | sed -e '/^$/,$d'
<span class="line-numbers"> 5</span>processor       : 0
<span class="line-numbers"> 6</span>vendor_id       : GenuineIntel
<span class="line-numbers"> 7</span>cpu family      : 6
<span class="line-numbers"> 8</span>model           : 62
<span class="line-numbers"> 9</span>model name      : Intel(R) Xeon(R) CPU E5-4627 v2 @ 3.30GHz   #&lt;------
<span class="line-numbers">10</span>stepping        : 4
<span class="line-numbers">11</span>microcode       : 0x427
<span class="line-numbers">12</span>cpu MHz         : 3299.797
<span class="line-numbers">13</span>cache size      : 16384 KB
<span class="line-numbers">14</span>physical id     : 0
<span class="line-numbers">15</span>siblings        : 8
<span class="line-numbers">16</span>core id         : 0
<span class="line-numbers">17</span>cpu cores       : 8
<span class="line-numbers">18</span>apicid          : 0
<span class="line-numbers">19</span>initial apicid  : 0
<span class="line-numbers">20</span>fpu             : yes
<span class="line-numbers">21</span>fpu_exception   : yes
<span class="line-numbers">22</span>cpuid level     : 13
<span class="line-numbers">23</span>wp              : yes
<span class="line-numbers">24</span>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr  <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">25</span>                  pge mca cmov pat pse36 clflush dts acpi mmx fxsr
<span class="line-numbers">26</span>                  sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp   <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">27</span>                  lm constant_tsc arch_perfmon pebs bts rep_good
<span class="line-numbers">28</span>                  nopl xtopology nonstop_tsc aperfmperf eagerfpu
<span class="line-numbers">29</span>                  pni pclmulqdq dtes64 monitor ds_cpl vmx smx est   <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">30</span>                  tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2
<span class="line-numbers">31</span>                  x2apic popcnt tsc_deadline_timer aes xsave avx
<span class="line-numbers">32</span>                  f16c rdrand lahf_lm ida arat epb pln pts dtherm   <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">33</span>                  tpr_shadow vnmi flexpriority ept vpid fsgsbase
<span class="line-numbers">34</span>                  smep erms xsaveopt
<span class="line-numbers">35</span>bugs            :
<span class="line-numbers">36</span>bogomips        : 6599.59
<span class="line-numbers">37</span>clflush size    : 64
<span class="line-numbers">38</span>cache_alignment : 64
<span class="line-numbers">39</span>address sizes   : 46 bits physical, 48 bits virtual
<span class="line-numbers">40</span>power management:</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td><code>pse</code> flag indicate 2M hugepage support</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td><code>pdpe1gb</code> flag indicate 1G hugepage support</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td><code>vmx</code> flag indicate VT-x capability</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td><code>rdrand</code> flag, required by current VMX implementation, supported in Ivy
Bridge CPU and not in Sandy Bridge CPU</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">memory</div>
<div class="content">
<pre>ping@trinity: free -h
             total       used       free     shared    buffers     cached
Mem:          503G        13G       490G       1.5M       176M       6.9G
-/+ buffers/cache:       5.9G       497G
Swap:          14G         0B        14G</pre>
</div>
</div>
<div class="listingblock">
<div class="title"><code>dmidecode</code> version of the cpu information captured is shown below:</div>
<div class="content">
<pre>ping@trinity: sudo dmidecode -t 4
# dmidecode 2.12
SMBIOS 2.8 present.

    Handle 0x0400, DMI type 4, 42 bytes
    Processor Information
            Socket Designation: Proc 1
            Type: Central Processor
            Family: Xeon
            Manufacturer: Intel
            ID: E4 06 03 00 FF FB EB BF
            Signature: Type 0, Family 6, Model 62, Stepping 4
            Flags:
                    FPU (Floating-point unit on-chip)
                    VME (Virtual mode extension)
                    DE (Debugging extension)
                    PSE (Page size extension)
                    TSC (Time stamp counter)
                    MSR (Model specific registers)
                    PAE (Physical address extension)
                    MCE (Machine check exception)
                    CX8 (CMPXCHG8 instruction supported)
                    APIC (On-chip APIC hardware supported)
                    SEP (Fast system call)
                    MTRR (Memory type range registers)
                    PGE (Page global enable)
                    MCA (Machine check architecture)
                    CMOV (Conditional move instruction supported)
                    PAT (Page attribute table)
                    PSE-36 (36-bit page size extension)
                    CLFSH (CLFLUSH instruction supported)
                    DS (Debug store)
                    ACPI (ACPI supported)
                    MMX (MMX technology supported)
                    FXSR (FXSAVE and FXSTOR instructions supported)
                    SSE (Streaming SIMD extensions)
                    SSE2 (Streaming SIMD extensions 2)
                    SS (Self-snoop)
                    HTT (Multi-threading)
                    TM (Thermal monitor supported)
                    PBE (Pending break enabled)
            Version:  Intel(R) Xeon(R) CPU E5-4627 v2 @ 3.30GHz
            Voltage: 1.4 V
            External Clock: 100 MHz
            Max Speed: 4800 MHz
            Current Speed: 3300 MHz
            Status: Populated, Enabled
            Upgrade: Socket LGA2011
            L1 Cache Handle: 0x0710
            L2 Cache Handle: 0x0720
            L3 Cache Handle: 0x0730
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: Not Specified
            Core Count: 8
            Core Enabled: 8
            Thread Count: 8
            Characteristics:
                    64-bit capable

    Handle 0x0401, DMI type 4, 42 bytes
    Processor Information
            Socket Designation: Proc 2
            Type: Central Processor
            ......

    Handle 0x0402, DMI type 4, 42 bytes
    Processor Information
            Socket Designation: Proc 3
            ......

    Handle 0x0403, DMI type 4, 42 bytes
    Processor Information
            Socket Designation: Proc 4
            ......</pre>
</div>
</div>
<div class="listingblock">
<div class="title"><code>dmidecode</code> version of the memory data is: <code>dmidecode -t 17</code>:</div>
<div class="content">
<pre>ping@trinity: sudo dmidecode -t 17
# dmidecode 2.12
SMBIOS 2.8 present.

    Handle 0x1100, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 32 GB         #&lt;------
            Form Factor: DIMM
            Set: None
            Locator: PROC  1 DIMM  1
            Bank Locator: Not Specified
            Type: DDR3
            Type Detail: Synchronous
            Speed: 1866 MHz
            Manufacturer: HP
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: 712384-081
            Rank: 4
            Configured Clock Speed: 1866 MHz
            Minimum voltage:  1.500 V
            Maximum voltage:  1.500 V
            Configured voltage:  1.500 V

    Handle 0x1101, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1000
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: No Module Installed
            Form Factor: DIMM
            Set: 1
            Locator: PROC  1 DIMM  2
            Bank Locator: Not Specified
            Type: DDR3
            Type Detail: Synchronous
            Speed: Unknown
            Manufacturer: UNKNOWN
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: NOT AVAILABLE
            Rank: Unknown
            Configured Clock Speed: Unknown
            Minimum voltage:  Unknown
            Maximum voltage:  Unknown
            Configured voltage:  Unknown

    Handle 0x1102, DMI type 17, 40 bytes
    Memory Device
            ......
            Set: 2
            Locator: PROC  1 DIMM  3
            ......

    ......

    Handle 0x111F, DMI type 17, 40 bytes
    Memory Device
            Array Handle: 0x1003
            Error Information Handle: Not Provided
            Total Width: 72 bits
            Data Width: 64 bits
            Size: 32 GB
            Form Factor: DIMM
            Set: 31
            Locator: PROC  4 DIMM  8
            Bank Locator: Not Specified
            Type: DDR3
            Type Detail: Synchronous
            Speed: 1866 MHz
            Manufacturer: HP
            Serial Number: Not Specified
            Asset Tag: Not Specified
            Part Number: 712384-081
            Rank: 4
            Configured Clock Speed: 1866 MHz
            Minimum voltage:  1.500 V
            Maximum voltage:  1.500 V
            Configured voltage:  1.500 V</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_network_adapter_controller"><a class="anchor" href="#_network_adapter_controller"></a>1.1.4. network adapter/controller</h4>
<div class="ulist">
<div class="title">This server&#8217;s adapter/NIC/controller info:</div>
<ul>
<li>
<p>two type of HP NIC adapters were equiped:</p>
<div class="ulist">
<ul>
<li>
<p><a href="http://www8.hp.com/us/en/products/iss-adapters/product-detail.html?oid=5274097">HP 560FLB adapter</a>,</p>
</li>
<li>
<p>HP 560M adapter,</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="http://www.intel.com/content/www/us/en/embedded/products/networking/82599-10-gbe-controller-datasheet.html">Intel 82599 10GE controller</a></p>
</li>
<li>
<p>ixgbe kernel driver module that came with linux kernel is in use</p>
<div class="ulist">
<ul>
<li>
<p>SR-IOV supported</p>
</li>
<li>
<p>by default VF has not yet been configured</p>
</li>
</ul>
</div>
</li>
<li>
<p>adapter/interface/PCI address mapping table of all physical network
interfaces in this server:</p>
<table class="tableblock frame-all grid-all" style="width: 50%;">
<caption class="title">Table 1. server interface table</caption>
<colgroup>
<col style="width: 15%;">
<col style="width: 30%;">
<col style="width: 30%;">
<col style="width: 23%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">NO.</th>
<th class="tableblock halign-left valign-top">"PCI address"</th>
<th class="tableblock halign-left valign-top">adapter</th>
<th class="tableblock halign-left valign-top">interface name</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em9</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em10</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p2p1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p2p2</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em2</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p3p1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">8</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p3p2</p></td>
</tr>
</tbody>
</table>
</li>
</ul>
</div>
<div class="paragraph">
<p><a id="lspci"></a>[lspci]
<code>lspci</code> is a linux utility for displaying information about PCI buses in the
system and devices connected to them.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ lspci | grep -i ethernet
<span class="line-numbers"> 2</span>02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 3</span>02:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 4</span>06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 5</span>06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 6</span>21:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 7</span>21:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 8</span>23:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 9</span>23:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)</code></pre>
</div>
</div>
<div class="paragraph">
<p>the prefix number string of each line is a "PCI address":</p>
</div>
<div class="listingblock">
<div class="content">
<pre>02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
 ^  ^ ^
 |  | |
 |  | function (port): 0-7
 |  |
 | slot (NIC): 0-1f
 |
bus: 0-ff</pre>
</div>
</div>
<div class="paragraph">
<p>there is also a "domain" before "bus:slot.function", usually with a value
<code>0000</code> and is ignored.  see <code>man lspci</code> option <code>-s</code> and <code>-D</code>.</p>
</div>
<div class="listingblock">
<div class="title">to get more details of each device (NIC port) via PCI address:</div>
<div class="content">
<pre>ping@trinity:~$ sudo lspci -vs 02:00.0
02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01) <i class="conum" data-value="2"></i><b>(2)</b>
    Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560FLB Adapter <i class="conum" data-value="1"></i><b>(1)</b>
    Flags: bus master, fast devsel, latency 0, IRQ 64
    Memory at ef700000 (32-bit, non-prefetchable) [size=1M]
    I/O ports at 4000 [size=32]
    Memory at ef6f0000 (32-bit, non-prefetchable) [size=16K]
    [virtual] Expansion ROM at ef400000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)            <i class="conum" data-value="4"></i><b>(4)</b>
    Kernel driver in use: ixgbe                                            <i class="conum" data-value="3"></i><b>(3)</b>

ping@trinity:~$ sudo lspci -vs 06:00.0
06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01) <i class="conum" data-value="2"></i><b>(2)</b>
    Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560M Adapter <i class="conum" data-value="1"></i><b>(1)</b>
    Physical Slot: 2
    Flags: bus master, fast devsel, latency 0, IRQ 136
    Memory at eff00000 (32-bit, non-prefetchable) [size=1M]
    I/O ports at 6000 [size=32]
    Memory at efef0000 (32-bit, non-prefetchable) [size=16K]
    [virtual] Expansion ROM at efc00000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)          <i class="conum" data-value="4"></i><b>(4)</b>
    Kernel driver in use: ixgbe                                          <i class="conum" data-value="3"></i><b>(3)</b>

ping@trinity:~$ sudo lspci -vs 21:* | grep -iE "controller|adapter"
21:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01) <i class="conum" data-value="2"></i><b>(2)</b>
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560FLB Adapter <i class="conum" data-value="1"></i><b>(1)</b>
21:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560FLB Adapter

ping@trinity:~$ sudo lspci -vs 23:* | grep -iE "controller|adapter"
23:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560M Adapter
23:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560M Adapter</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>adapter vendor info</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>controller vendor info</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>current driver in use is ixgbe</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>driver kernel module support SR-IOV feature</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_ixgbe_kernel_driver"><a class="anchor" href="#_ixgbe_kernel_driver"></a>1.1.5. ixgbe kernel driver</h4>
<div class="ulist">
<ul>
<li>
<p>current ixgbe driver of this server is the one that came with linux kernel
<code>3.19.1</code>:</p>
<div class="ulist">
<ul>
<li>
<p>ixgbe version 4.0.1-k</p>
</li>
<li>
<p>support following capabilities:</p>
<div class="ulist">
<ul>
<li>
<p><code>max_vfs</code> parameter: max 63 VF allowed per port (default 0: VF won&#8217;t be
enabled by default)</p>
</li>
<li>
<p><code>allow_unsupported_sfp</code> parameter: unsupported/untested SFP allowed</p>
</li>
<li>
<p><code>debug</code> option</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>the <a href="#folder-structure">README.txt</a> file from VMX installation package
contains more info about the ixgbe driver</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ cat /sys/module/ixgbe/version
4.0.1-k</pre>
</div>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ modinfo ixgbe
<span class="line-numbers"> 2</span>filename:
<span class="line-numbers"> 3</span>  /lib/modules/3.19.0-25-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
<span class="line-numbers"> 4</span>version:        4.0.1-k     #&lt;------
<span class="line-numbers"> 5</span>license:        GPL
<span class="line-numbers"> 6</span>description:    Intel(R) 10 Gigabit PCI Express Network Driver
<span class="line-numbers"> 7</span>author:         Intel Corporation, &lt;linux.nics@intel.com&gt;
<span class="line-numbers"> 8</span>srcversion:     44CBFE422F8BAD726E61653
<span class="line-numbers"> 9</span>alias:          pci:v00008086d000015ABsv*sd*bc*sc*i*
<span class="line-numbers">10</span>alias:          pci:v00008086d000015AAsv*sd*bc*sc*i*
<span class="line-numbers">11</span>alias:          pci:v00008086d00001563sv*sd*bc*sc*i*
<span class="line-numbers">12</span>alias:          pci:v00008086d00001560sv*sd*bc*sc*i*
<span class="line-numbers">13</span>alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*
<span class="line-numbers">14</span>alias:          pci:v00008086d00001557sv*sd*bc*sc*i*
<span class="line-numbers">15</span>alias:          pci:v00008086d00001558sv*sd*bc*sc*i*
<span class="line-numbers">16</span>alias:          pci:v00008086d0000154Fsv*sd*bc*sc*i*
<span class="line-numbers">17</span>alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*
<span class="line-numbers">18</span>alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
<span class="line-numbers">19</span>alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*
<span class="line-numbers">20</span>alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*
<span class="line-numbers">21</span>alias:          pci:v00008086d00001529sv*sd*bc*sc*i*
<span class="line-numbers">22</span>alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*
<span class="line-numbers">23</span>alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*
<span class="line-numbers">24</span>alias:          pci:v00008086d00001514sv*sd*bc*sc*i*
<span class="line-numbers">25</span>alias:          pci:v00008086d00001507sv*sd*bc*sc*i*
<span class="line-numbers">26</span>alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*
<span class="line-numbers">27</span>alias:          pci:v00008086d00001517sv*sd*bc*sc*i*
<span class="line-numbers">28</span>alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*
<span class="line-numbers">29</span>alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*
<span class="line-numbers">30</span>alias:          pci:v00008086d00001508sv*sd*bc*sc*i*
<span class="line-numbers">31</span>alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*
<span class="line-numbers">32</span>alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*
<span class="line-numbers">33</span>alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*
<span class="line-numbers">34</span>alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*
<span class="line-numbers">35</span>alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*
<span class="line-numbers">36</span>alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*
<span class="line-numbers">37</span>alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*
<span class="line-numbers">38</span>alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*
<span class="line-numbers">39</span>alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*
<span class="line-numbers">40</span>alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*
<span class="line-numbers">41</span>alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
<span class="line-numbers">42</span>depends:        mdio,ptp,dca,vxlan
<span class="line-numbers">43</span>intree:         Y
<span class="line-numbers">44</span>vermagic:       3.19.0-25-generic SMP mod_unload modversions
<span class="line-numbers">45</span>signer:         Magrathea: Glacier signing key
<span class="line-numbers">46</span>sig_key:        6A:AA:11:D1:8C:2D:3A:40:B1:B4:DB:E5:BF:8A:D6:56:DD:F5:18:38
<span class="line-numbers">47</span>sig_hashalgo:   sha512
<span class="line-numbers">48</span>parm:           max_vfs:Maximum number of virtual functions to allocate per
<span class="line-numbers">49</span>    physical function - default is zero and maximum value is 63. (Deprecated)
<span class="line-numbers">50</span>    (uint)
<span class="line-numbers">51</span>parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+
<span class="line-numbers">52</span>    modules on 82599-based adapters (uint)
<span class="line-numbers">53</span>parm:           debug:Debug level (0=none,...,16=all) (int)</code></pre>
</div>
</div>
<div class="ulist">
<div class="title">reference</div>
<ul>
<li>
<p><a href="http://www.intel.com/content/dam/doc/design-guide/82599-sr-iov-driver-companion-guide.pdf">Intel® 82599 SR-IOV Driver Companion Guide</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="IOMMU"><a class="anchor" href="#IOMMU"></a>1.1.6. VT-d/IOMMU</h4>
<div class="paragraph">
<p>Intel <code>VT-d</code> feaure is supported in this server&#8217;s current OS kernel.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ less /boot/config-3.19.0-25-generic | grep -i iommu
<span class="line-numbers"> 2</span>CONFIG_GART_IOMMU=y
<span class="line-numbers"> 3</span>CONFIG_CALGARY_IOMMU=y
<span class="line-numbers"> 4</span>CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
<span class="line-numbers"> 5</span>CONFIG_IOMMU_HELPER=y
<span class="line-numbers"> 6</span>CONFIG_VFIO_IOMMU_TYPE1=m
<span class="line-numbers"> 7</span>CONFIG_IOMMU_API=y
<span class="line-numbers"> 8</span>CONFIG_IOMMU_SUPPORT=y          #&lt;------
<span class="line-numbers"> 9</span>CONFIG_AMD_IOMMU=y
<span class="line-numbers">10</span>CONFIG_AMD_IOMMU_STATS=y
<span class="line-numbers">11</span>CONFIG_AMD_IOMMU_V2=m
<span class="line-numbers">12</span>CONFIG_INTEL_IOMMU=y            #&lt;------
<span class="line-numbers">13</span># CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
<span class="line-numbers">14</span>CONFIG_INTEL_IOMMU_FLOPPY_WA=y
<span class="line-numbers">15</span># CONFIG_IOMMU_DEBUG is not set
<span class="line-numbers">16</span># CONFIG_IOMMU_STRESS is not set
<span class="line-numbers">17</span>
<span class="line-numbers">18</span>ping@trinity:~$ grep -i remap /boot/config-3.13.0-32-generic
<span class="line-numbers">19</span>CONFIG_HAVE_IOREMAP_PROT=y
<span class="line-numbers">20</span>CONFIG_IRQ_REMAP=y              #&lt;------
<span class="line-numbers">21</span>
<span class="line-numbers">22</span>ping@ubuntu1:~$ grep -i pci_stub /boot/config-3.13.0-32-generic
<span class="line-numbers">23</span>CONFIG_PCI_STUB=m               #&lt;------</code></pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
a more "general" way to print kernel info is: <code>less /boot/config-`uname -r`</code>
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_kvm_qemu_software_version"><a class="anchor" href="#_kvm_qemu_software_version"></a>1.1.7. kvm/qemu software version</h4>
<div class="ulist">
<ul>
<li>
<p>qemu version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.19)</p>
</li>
<li>
<p>kvm version (same as linux kernel version)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This looks OK comparing with the <a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/reference/general/vmx-hw-sw-minimums.html">"Minimum Hardware and
Software"</a> from the release note:</p>
</div>
<div class="quoteblock">
<blockquote>
VirtualizationQEMU-KVM 2.0.0+dfsg-2ubuntu1.11 or later
</blockquote>
</div>
<div class="paragraph">
<p>To verify qemu/kvm version and hardware acceleration support:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>qemu version:</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ qemu-system-x86_64 --version
<span class="line-numbers">2</span>QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.19), Copyright (c) 2003-2008 Fabrice Bellard</code></pre>
</div>
</div>
<div class="paragraph">
<p>or</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ kvm --version
<span class="line-numbers">2</span>QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.19), Copyright (c) 2003-2008 Fabrice Bellard</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>KVM kernel module info:</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ modinfo kvm
<span class="line-numbers"> 2</span>filename:       /lib/modules/3.19.0-25-generic/kernel/arch/x86/kvm/kvm.ko
<span class="line-numbers"> 3</span>license:        GPL
<span class="line-numbers"> 4</span>author:         Qumranet
<span class="line-numbers"> 5</span>srcversion:     F58A0F8858A02EFA0549DE5
<span class="line-numbers"> 6</span>depends:
<span class="line-numbers"> 7</span>intree:         Y
<span class="line-numbers"> 8</span>vermagic:       3.19.0-25-generic SMP mod_unload modversions
<span class="line-numbers"> 9</span>signer:         Magrathea: Glacier signing key
<span class="line-numbers">10</span>sig_key:        6A:AA:11:D1:8C:2D:3A:40:B1:B4:DB:E5:BF:8A:D6:56:DD:F5:18:38
<span class="line-numbers">11</span>sig_hashalgo:   sha512
<span class="line-numbers">12</span>parm:           allow_unsafe_assigned_interrupts:Enable device assignment on platforms without interrupt remapping support. (bool)
<span class="line-numbers">13</span>parm:           ignore_msrs:bool
<span class="line-numbers">14</span>parm:           min_timer_period_us:uint
<span class="line-numbers">15</span>parm:           tsc_tolerance_ppm:uint</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>HW acceleration OK:</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

ping@MX86-host-BL660C-B1: ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Nov  5 11:35 /dev/kvm</pre>
</div>
</div>
</div>
</div>
</li>
</ol>
</div>
<div class="sidebarblock">
<div class="content">
<div class="paragraph">
<p>The development of qemu software is very active, and code changes (for bug fix,
security update, new features) and new releases appear frequently. For ubuntu
linux the changelogs are available
<a href="https://launchpad.net/ubuntu/trusty/+source/qemu/+changelog">here</a></p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_libvirt"><a class="anchor" href="#_libvirt"></a>1.1.8. libvirt</h4>
<div class="ulist">
<ul>
<li>
<p>current libvirt version is 1.2.2</p>
</li>
<li>
<p>this needs to be upgraded to <a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/task/installation/vmx-install-preparing.html#libvirt-upgrade">required libvirt</a> version.</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@ubuntu:~$ libvirtd --version
<span class="line-numbers">2</span>libvirtd (libvirt) 1.2.2</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
libvirt 1.2.2 misses some of bug fixes of features that are important for
VMX.  One of these are <code>numatune</code>, which is used to "pin" vCPUs of guest VM to
physcial CPUs all in one NUMA node, hence reducing the <code>NUMA misses</code> that is
one of the main contributor of performance impact.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Now after identifying the server HW/SW configuration, the checklist looks:</p>
</div>
<div class="ulist checklist">
<div class="title">pre-installation checklist</div>
<ul class="checklist">
<li>
<p><i class="fa fa-square-o"></i> server manufacturer info</p>
</li>
<li>
<p><i class="fa fa-check-square-o"></i> operating system</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> cpu and memory</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> NIC/controller</p>
</li>
<li>
<p><i class="fa fa-check-square-o"></i> NIC driver</p>
</li>
<li>
<p><i class="fa fa-check-square-o"></i> VT-d/IOMMU</p>
</li>
<li>
<p><i class="fa fa-square-o"></i> kvm/qemu version</p>
</li>
<li>
<p><i class="fa fa-check-square-o"></i> libvirt version</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>those checked mark indicate those items that do not meet the requirement and
need some change.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_adjust_the_system"><a class="anchor" href="#_adjust_the_system"></a>1.2. adjust the system</h3>
<div class="paragraph">
<p>after collecting the server&#8217;s current configuration, we need to change some
setting according to the
<a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/task/installation/vmx-install-preparing.html">Preparing
the System to Install vMX</a> portion in the VMX document.</p>
</div>
<div class="sect3">
<h4 id="_bios_setting"><a class="anchor" href="#_bios_setting"></a>1.2.1. BIOS setting</h4>
<div class="paragraph">
<p>following (Intel) virtualization features are required to setup VMX and need to
be enabled from within BIOS, if not yet.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>VT-x</p>
</li>
<li>
<p>VT-d</p>
</li>
<li>
<p>SR-IOV</p>
</li>
<li>
<p>HyperThreading</p>
</li>
</ul>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<i class="fa icon-important" title="Important"></i>
</td>
<td class="content">
the requirment may change in different VMX release
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>enter BIOS setting:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/11860415/865494a4-a442-11e5-8e62-05c126a9b004.png" alt="865494a4 a442 11e5 8e62 05c126a9b004">
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>even in same hardware, different version of BIOS might look a little bit
different. In this system we have BIOS version "I32"</p>
</div>
<div class="paragraph">
<p>Here are more details of current BIOS info in this server:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>labroot@MX86-host-BL660C-B1:~/vmx_20141216$ sudo dmidecode
# dmidecode 2.12
SMBIOS 2.8 present.
227 structures occupying 7860 bytes.
Table at 0xBFBDB000.</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: HP
        Version: I32        #&lt;------
        Release Date: 02/10/2014
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 8192 kB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                ESCD support is available
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                5.25"/360 kB floppy services are supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
        Firmware Revision: 1.51</pre>
</div>
</div>
</td>
</tr>
</table>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>VT-x/VT-d/HyperThreading</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p><span class="menuseq"><span class="menu">BIOS</span>&#160;&#9656; <span class="submenu">System Options</span>&#160;&#9656; <span class="menuitem">Intel&#174; Virtualization technology</span></span></p>
</div>
<div class="paragraph">
<p><span class="menuseq"><span class="menu">BIOS</span>&#160;&#9656; <span class="submenu">System Options</span>&#160;&#9656; <span class="menuitem">Intel&#174; VT-d</span></span></p>
</div>
<div class="paragraph">
<p><span class="menuseq"><span class="menu">BIOS</span>&#160;&#9656; <span class="submenu">System Options</span>&#160;&#9656; <span class="menuitem">Intel&#174; HyperThreading Options</span></span></p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/11860455/04fc70ec-a443-11e5-9ce0-f02a3a05a7c1.png" alt="04fc70ec a443 11e5 9ce0 f02a3a05a7c1">
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>The <a href="http://www.juniper.net/techpubs/en_US/vmx15.1/information-products/topic-collections/release-notes/index.html">new VMX</a> releases requires HyperThreading to be enabled
to support flow cache feature. The installation script will abort if HT is not
enabled, see <a href="#troubleshooting-install">troubleshooting installation script</a>
for more detail of this issue. To install VMX, Either modifying the script to
disable HT1 calculation/verification, or installing VMX manually.</p>
</div>
<div class="paragraph">
<p>According to <a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/reference/general/vmx-hw-sw-minimums.html">"Minimum Hardware and Software Requirements"</a>:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="ulist">
<ul>
<li>
<p>For lab simulation and low performance (less than 100 Mbps) use cases, any
x86 processor (Intel or AMD) with VT-d capability.</p>
</li>
<li>
<p>For all other use cases, Intel Ivy Bridge processors or later are required.
Example of Ivy Bridge processor: Intel Xeon E5-2667 v2 @ 3.30 GHz 25 MB Cache</p>
</li>
<li>
<p>For single root I/O virtualization (SR-IOV) NIC type, use Intel 82599-based
PCI-Express cards (10 Gbps) and Ivy Bridge processors.</p>
</li>
</ul>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>These statements indicate some current implementation info:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>lab simulation/low performance &#8658; virtio</p>
</li>
<li>
<p>all other use cases &#8658; high performance &#8658; SRIOV</p>
</li>
<li>
<p>"Ivy Bridge CPU" is required for VMX running SR-IOV</p>
</li>
</ul>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</li>
<li>
<p>SR-IOV</p>
<div class="paragraph">
<p><span class="menuseq"><span class="menu">BIOS</span>&#160;&#9656; <span class="submenu">Advanced Options</span>&#160;&#9656; <span class="menuitem">SR-IOV</span></span></p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/11860437/da2d5aa2-a442-11e5-9ad7-837bfe167b20.png" alt="da2d5aa2 a442 11e5 9ad7 837bfe167b20">
</div>
</div>
</li>
</ol>
</div>
</div>
<div class="sect3">
<h4 id="enable-iommu"><a class="anchor" href="#enable-iommu"></a>1.2.2. enable iommu/VT-d</h4>
<div class="paragraph">
<p>to enable VT-d we need to change kernel boot parameters. Here is the steps:</p>
</div>
<div class="openblock">
<div class="content">
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Make sure the <code>/etc/default/grub</code> file contains this line:</p>
<div class="literalblock">
<div class="content">
<pre>GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"</pre>
</div>
</div>
</li>
<li>
<p>If not, add it:</p>
<div class="literalblock">
<div class="content">
<pre>echo 'GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"' &gt;&gt; /etc/default/grub</pre>
</div>
</div>
<div class="paragraph">
<p>So it looks like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ grep -i iommu /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"</pre>
</div>
</div>
</li>
<li>
<p>Run <code>sudo update-grub</code> to update grub</p>
</li>
<li>
<p>reboot system to make it start with configured kernel parameters</p>
<div class="literalblock">
<div class="content">
<pre>sudo reboot</pre>
</div>
</div>
</li>
</ol>
</div>
</div>
</div>
<div class="admonitionblock caution">
<table>
<tr>
<td class="icon">
<i class="fa icon-caution" title="Caution"></i>
</td>
<td class="content">
when using 'echo', be careful to use <code>&gt;&gt;</code> instead of <code>&gt;</code>. <code>&gt;</code> will
"overwrite" the whole file with whatever echoed, instead of "append". in case
that happens, correct it with
<a href="http://askubuntu.com/questions/509923/whats-the-default-14-04-etc-default-grub-file-contents">instruction here</a>.
</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>instead of checking the grub config file, looking at parameters passed to the
kernel at the time it is started might be more accurate:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@matrix:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.19.0-25-generic.efi.signed root=UUID=875bb72c-c5de-4329-af48-55af85f26398 ro intel_iommu=on pci=realloc</pre>
</div>
</div>
<div class="paragraph">
<p>in this example, we are sure the current kernel has IOMMU enabled.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>In a system that didn&#8217;t enable IOMMU, after enabling it and restart, you will
notice kernel logs similar to below captures:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@Compute24:~$ dmesg -T | grep -iE "iommu|dmar"
[Thu Dec 17 09:07:22 2015] Command line: BOOT_IMAGE=/vmlinuz-3.16.0-30-generic
    root=/dev/mapper/Compute24--vg-root ro intel_iommu=on
    crashkernel=384M-2G:64M,2G-16G:128M,16G-:256M
[Thu Dec 17 09:07:22 2015] ACPI: DMAR 0x00000000BDDAB840 000718 (v01 HP
    ProLiant 00000001 \xffffffd2?   0000162E)
[Thu Dec 17 09:07:22 2015] Kernel command line:
    BOOT_IMAGE=/vmlinuz-3.16.0-30-generic root=/dev/mapper/Compute24--vg-root ro
    intel_iommu=on crashkernel=384M-2G:64M,2G-16G:128M,16G-:256M
[Thu Dec 17 09:07:22 2015] Intel-IOMMU: enabled
[Thu Dec 17 09:07:22 2015] dmar: Host address width 46
[Thu Dec 17 09:07:22 2015] dmar: DRHD base: 0x000000f34fe000 flags: 0x0
[Thu Dec 17 09:07:22 2015] dmar: IOMMU 0: reg_base_addr f34fe000 ver 1:0 cap d2078c106f0466 ecap f020de
[Thu Dec 17 09:07:22 2015] dmar: DRHD base: 0x000000f7efe000 flags: 0x0
[Thu Dec 17 09:07:22 2015] dmar: IOMMU 1: reg_base_addr f7efe000 ver 1:0 cap d2078c106f0466 ecap f020de
[Thu Dec 17 09:07:22 2015] dmar: DRHD base: 0x000000fbefe000 flags: 0x0
[Thu Dec 17 09:07:22 2015] dmar: IOMMU 2: reg_base_addr fbefe000 ver 1:0 cap d2078c106f0466 ecap f020de
[Thu Dec 17 09:07:22 2015] dmar: DRHD base: 0x000000ecffe000 flags: 0x1
[Thu Dec 17 09:07:22 2015] dmar: IOMMU 3: reg_base_addr ecffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[Thu Dec 17 09:07:22 2015] dmar: RMRR base: 0x000000bdffd000 end: 0x000000bdffffff
......
[Thu Dec 17 09:07:22 2015] dmar: RMRR base: 0x000000bddde000 end: 0x000000bdddefff
[Thu Dec 17 09:07:22 2015] dmar: ATSR flags: 0x0
[Thu Dec 17 09:07:22 2015] IOAPIC id 12 under DRHD base  0xfbefe000 IOMMU 2
[Thu Dec 17 09:07:22 2015] IOAPIC id 11 under DRHD base  0xf7efe000 IOMMU 1
[Thu Dec 17 09:07:22 2015] IOAPIC id 10 under DRHD base  0xf34fe000 IOMMU 0
[Thu Dec 17 09:07:22 2015] IOAPIC id 8 under DRHD base  0xecffe000 IOMMU 3
[Thu Dec 17 09:07:22 2015] IOAPIC id 0 under DRHD base  0xecffe000 IOMMU 3
[Thu Dec 17 09:07:24 2015] IOMMU 2 0xfbefe000: using Queued invalidation
[Thu Dec 17 09:07:24 2015] IOMMU 1 0xf7efe000: using Queued invalidation
[Thu Dec 17 09:07:24 2015] IOMMU 0 0xf34fe000: using Queued invalidation
[Thu Dec 17 09:07:24 2015] IOMMU 3 0xecffe000: using Queued invalidation
[Thu Dec 17 09:07:24 2015] IOMMU: Setting RMRR:
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:01:00.0 [0xbddde000 - 0xbdddefff]
......
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:01:00.2 [0xbddde000 - 0xbdddefff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:01:00.4 [0xbddde000 - 0xbdddefff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:02:00.1 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:06:00.0 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:06:00.1 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:21:00.0 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:21:00.1 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:23:00.0 [0xe8000 - 0xe8fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:23:00.1 [0xe8000 - 0xe8fff]
......
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:23:00.0 [0xbdf83000 - 0xbdf84fff]
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:23:00.1 [0xbdf83000 - 0xbdf84fff]
......
[Thu Dec 17 09:07:24 2015] IOMMU: Prepare 0-16MiB unity mapping for LPC
[Thu Dec 17 09:07:24 2015] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="install-linux-kernel"><a class="anchor" href="#install-linux-kernel"></a>1.2.3. install required linux kernel for SR-IOV based VMX</h4>
<div class="paragraph">
<p>linux kernel needs to be changed if VMX needs to be setup with <code>SR-IOV</code>.</p>
</div>
<div class="paragraph">
<p>currently the ixgbe coming with ubuntu does not work on VMX. The main issue is
<strong>lack of multicast support on ingress - packet received on a VF will be
discarded siliently and won&#8217;t be delivered into the guest VM, an example of the
immediate effect of this is that OSPF (and most of today&#8217;s IGP) neighborship
won&#8217;t come up</strong>. Therefore building VMX based on <code>SR-IOV</code> requires to re-compile
the ixgbe kernel driver from source code, which is provided by Juniper to fix
the multicast support.  the code is available in the installation package. At
the time of the writing of this document there is problem to compile ixgbe from
source code under any kernels other than <code>3.13.0-32-generic</code>. that&#8217;s why the
kernel needs to be changed in this setup.</p>
</div>
<div class="paragraph">
<p>There is a statement about this issue in the VMX document:</p>
</div>
<div class="quoteblock">
<blockquote>
Modified IXGBE drivers are included in the package. Multicast promiscuous
mode for Virtual Functions is needed to receive control traffic that comes with
broadcast MAC addresses. The reference driver does not come with this mode set,
so the IXGBE drivers in this package contain certain modifications to overcome
this limitation.
</blockquote>
<div class="attribution">
&#8212; VMX "Getting Started Guide"
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
there is a plan to make ixgbe kernel module to work with newer kernel in
new VMX release.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>use below commands (provided in VMX installation doc) to change kernel:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install linux-firmware linux-image-3.13.0.32-generic \
                     linux-image-extra-3.13.0.32-generic \
                     linux-headers-3.13.0.32-generic</pre>
</div>
</div>
<div class="paragraph">
<div class="title">setup default linux kernel</div>
<p>After changing the kernel for the next reboot, you may want to make it default
kernel for later reboot. To achieve that following below steps:</p>
</div>
<div class="paragraph">
<p>in file <code>/boot/grub/grub.cfg</code> locate this line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>menuentry 'Ubuntu, with Linux 3.13.0-32-generic'</pre>
</div>
</div>
<div class="paragraph">
<p>then move it before the first <code>menuentry</code> entry:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>    export linux_gfx_mode
<span class="line-numbers">2</span>                                                #&lt;------move to here
<span class="line-numbers">3</span>    menuentry 'Ubuntu' --class ubuntu -  ....</code></pre>
</div>
</div>
<div class="paragraph">
<p>save and reboot the server.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
this is optional because otherwise you will still be given a change to
select kernel version during system reboot.
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_install_required_software_packages"><a class="anchor" href="#_install_required_software_packages"></a>1.2.4. install required software packages</h4>
<div class="paragraph">
<p>Use below commands (provided in VMX installation doc) to install required
software packages:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>sudo apt-get update
<span class="line-numbers">2</span>sudo apt-get install bridge-utils qemu-kvm libvirt-bin python numactl \
<span class="line-numbers">3</span>                    python-netifaces vnc4server libyaml-dev python-yaml\
<span class="line-numbers">4</span>                    libparted0-dev libpciaccess-dev libnuma-dev libyajl-dev\
<span class="line-numbers">5</span>                    libxml2-dev libglib2.0-dev libnl-dev libnl-dev python-pip\
<span class="line-numbers">6</span>                    python-dev libxml2-dev libxslt-dev</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>a quick way to verify if all/any of the required software package in the list
were installed correctly or not, is to simply re-run the above installation
commands again. If everything got installed correctly then you will get sth
like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>sudo apt-get install bridge-utils qemu-kvm libvirt-bin python numactl \
<span class="line-numbers"> 2</span>python-netifaces vnc4server libyaml-dev python-yaml libparted0-dev \
<span class="line-numbers"> 3</span>libpciaccess-dev libnuma-dev libyajl-dev libxml2-dev libglib2.0-dev \
<span class="line-numbers"> 4</span>libnl-dev libnl-dev python-pip python-dev libxml2-dev libxslt-dev
<span class="line-numbers"> 5</span>Reading package lists... Done
<span class="line-numbers"> 6</span>Building dependency tree
<span class="line-numbers"> 7</span>Reading state information... Done
<span class="line-numbers"> 8</span>Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
<span class="line-numbers"> 9</span>bridge-utils is already the newest version.
<span class="line-numbers">10</span>libpciaccess-dev is already the newest version.
<span class="line-numbers">11</span>libxslt1-dev is already the newest version.
<span class="line-numbers">12</span>libyajl-dev is already the newest version.
<span class="line-numbers">13</span>python is already the newest version.
<span class="line-numbers">14</span>python-dev is already the newest version.
<span class="line-numbers">15</span>python-netifaces is already the newest version.
<span class="line-numbers">16</span>libnl-dev is already the newest version.
<span class="line-numbers">17</span>libglib2.0-dev is already the newest version.
<span class="line-numbers">18</span>libnuma-dev is already the newest version.
<span class="line-numbers">19</span>libparted0-dev is already the newest version.
<span class="line-numbers">20</span>libvirt-bin is already the newest version.
<span class="line-numbers">21</span>libxml2-dev is already the newest version.
<span class="line-numbers">22</span>libyaml-dev is already the newest version.
<span class="line-numbers">23</span>python-yaml is already the newest version.
<span class="line-numbers">24</span>qemu-kvm is already the newest version.
<span class="line-numbers">25</span>numactl is already the newest version.
<span class="line-numbers">26</span>python-pip is already the newest version.
<span class="line-numbers">27</span>vnc4server is already the newest version.
<span class="line-numbers">28</span>0 upgraded, 0 newly installed, 0 to remove and 124 not upgraded.  #&lt;------</code></pre>
</div>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_libvirt_2"><a class="anchor" href="#_libvirt_2"></a>1.2.5. libvirt</h4>
<div class="paragraph">
<p>make sure libvirt version
<a href="https://www.redhat.com/archives/libvir-list/2014-September/msg00055.html">1.2.8</a>
is installed for "performance version" of VMX. refer to the VMX document for
detail steps to install it from source code.</p>
</div>
<div class="paragraph">
<p>In below command captures we demonstrate the
<a href="http://www.juniper.net/techpubs/en_US/vmx15.1/topics/task/installation/vmx-install-preparing.html#libvirt-upgrade">libvirt
upgrading process</a>.</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>original libvirt coming with ubuntu14.02:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@ubuntu:~$ libvirtd --version
<span class="line-numbers">2</span>libvirtd (libvirt) 1.2.2</code></pre>
</div>
</div>
</li>
<li>
<p>download and prepare source code:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>cd /tmp
<span class="line-numbers">2</span>wget http://libvirt.org/sources/libvirt-1.2.8.tar.gz
<span class="line-numbers">3</span>tar zxvf libvirt-1.2.8.tar.gz</code></pre>
</div>
</div>
</li>
<li>
<p>stop and uninstall old version:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>cd libvirt-1.2.8
<span class="line-numbers"> 2</span>sudo ./configure --prefix=/usr/local --with-numactl
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>    checking for a BSD-compatible install... /usr/bin/install -c
<span class="line-numbers"> 5</span>    checking whether build environment is sane... yes
<span class="line-numbers"> 6</span>    checking for a thread-safe mkdir -p... /bin/mkdir -p
<span class="line-numbers"> 7</span>    checking for gawk... gawk
<span class="line-numbers"> 8</span>    checking whether make sets $(MAKE)... yes
<span class="line-numbers"> 9</span>    ......
<span class="line-numbers">10</span>
<span class="line-numbers">11</span>sudo service libvirt-bin stop
<span class="line-numbers">12</span>    libvirt-bin stop/waiting
<span class="line-numbers">13</span>
<span class="line-numbers">14</span>sudo make uninstall
<span class="line-numbers">15</span>    Making uninstall in .
<span class="line-numbers">16</span>    make[1]: Entering directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">17</span>    make[1]: Leaving directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">18</span>    ......
<span class="line-numbers">19</span>
<span class="line-numbers">20</span>/bin/rm rf /usr/local/lib/libvirt*in/rm:
<span class="line-numbers">21</span>    cannot remove ‘/usr/local/lib/libvirt*’: No such file or directory</code></pre>
</div>
</div>
</li>
<li>
<p>install new version:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>sudo ./configure --prefix=/usr --localstatedir=/ --with-numactl
<span class="line-numbers"> 2</span>    checking for a BSD-compatible install... /usr/bin/install -c
<span class="line-numbers"> 3</span>    checking whether build environment is sane... yes
<span class="line-numbers"> 4</span>    checking for a thread-safe mkdir -p... /bin/mkdir -p
<span class="line-numbers"> 5</span>......
<span class="line-numbers"> 6</span>
<span class="line-numbers"> 7</span>sudo make
<span class="line-numbers"> 8</span>    make  all-recursive
<span class="line-numbers"> 9</span>    make[1]: Entering directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">10</span>    Making all in .
<span class="line-numbers">11</span>    make[2]: Entering directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">12</span>    make[2]: Leaving directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">13</span>    Making all in gnulib/lib
<span class="line-numbers">14</span>    make[2]: Entering directory `/tmp/libvirt-1.2.8/gnulib/lib'
<span class="line-numbers">15</span>      GEN      alloca.h
<span class="line-numbers">16</span>      GEN      c++defs.h
<span class="line-numbers">17</span>      GEN      warn-on-use.h
<span class="line-numbers">18</span>      GEN      arg-nonnull.h
<span class="line-numbers">19</span>      GEN      arpa/inet.h
<span class="line-numbers">20</span>    ......
<span class="line-numbers">21</span>
<span class="line-numbers">22</span>sudo make install
<span class="line-numbers">23</span>    Making install in .
<span class="line-numbers">24</span>    make[1]: Entering directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">25</span>    make[2]: Entering directory `/tmp/libvirt-1.2.8'
<span class="line-numbers">26</span>    ......</code></pre>
</div>
</div>
</li>
<li>
<p>start new version:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@ubuntu:/tmp/libvirt-1.2.8$ sudo service libvirt-bin start
<span class="line-numbers">2</span>libvirt-bin start/running, process 24450
<span class="line-numbers">3</span>ping@ubuntu:/tmp/libvirt-1.2.8$ ps aux| grep libvirt
<span class="line-numbers">4</span>
<span class="line-numbers">5</span>ping@ubuntu:/tmp/libvirt-1.2.8$ ps aux| grep libvirt
<span class="line-numbers">6</span>root     24450  0.5  0.0 405252 10772 ?        Sl   21:40   0:00 /usr/sbin/libvirtd -d</code></pre>
</div>
</div>
</li>
<li>
<p>verify the new version:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ libvirtd --version
<span class="line-numbers"> 2</span>libvirtd (libvirt) 1.2.8
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>ping@trinity:~$ service libvirt-bin status
<span class="line-numbers"> 5</span>libvirt-bin start/running, process 1559
<span class="line-numbers"> 6</span>
<span class="line-numbers"> 7</span>ping@trinity:~$ which libvirtd
<span class="line-numbers"> 8</span>/usr/sbin/libvirtd
<span class="line-numbers"> 9</span>
<span class="line-numbers">10</span>ping@trinity:~$ /usr/sbin/libvirtd --version
<span class="line-numbers">11</span>/usr/sbin/libvirtd (libvirt) 1.2.8
<span class="line-numbers">12</span>
<span class="line-numbers">13</span>ping@trinity:~$ virsh --version
<span class="line-numbers">14</span>1.2.8
<span class="line-numbers">15</span>
<span class="line-numbers">16</span>ping@trinity:/images/vmx_20151102.0$ sudo virsh -c qemu:///system version
<span class="line-numbers">17</span>Compiled against library: libvirt 1.2.8
<span class="line-numbers">18</span>Using library: libvirt 1.2.8
<span class="line-numbers">19</span>Using API: QEMU 1.2.8
<span class="line-numbers">20</span>Running hypervisor: QEMU 2.0.0</code></pre>
</div>
</div>
</li>
</ol>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_download_vmx_installation_package"><a class="anchor" href="#_download_vmx_installation_package"></a>1.3. download vmx installation package</h3>
<div class="paragraph">
<div class="title">locate the vmx tarball</div>
<p>download <a href="#folder-structure">VMX tarball</a> from internal or public server.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.juniper.net/support/downloads/?p=vmx">public URL</a></p>
</li>
<li>
<p>internal server:</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>pings@svl-jtac-tool02:/volume/publish/dev/wrlinux/mx86/15.1F_att_drop$ ls -l
<span class="line-numbers">2</span>total 4180932
<span class="line-numbers">3</span>-rw-rw-r--  1 rbu-builder  rbu-builder   816737275 Nov  9 12:23 vmx_20151102.0.tgz  #&lt;------
<span class="line-numbers">4</span>-rw-rw-r--  1 rbu-builder  rbu-builder  3447736320 Nov  4 12:18 vmx_20151102.0_tarball_issue.tgz</code></pre>
</div>
</div>
<div class="paragraph">
<p>To have a quick look at the guest image files included in the tarball:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>pings@svl-jtac-tool02:~$ cd /volume/publish/dev/wrlinux/mx86/15.1F_att_drop/
pings@svl-jtac-tool02:~$ tar tf vmx_20151102.0.tgz | grep images
vmx_20151102.0/images/
vmx_20151102.0/images/jinstall64-vmx-15.1F-20151104.0-domestic.img      <i class="conum" data-value="1"></i><b>(1)</b>
vmx_20151102.0/images/vFPC-20151102.img                                 <i class="conum" data-value="2"></i><b>(2)</b>
vmx_20151102.0/images/vmxhdd.img                                        <i class="conum" data-value="3"></i><b>(3)</b>
vmx_20151102.0/images/metadata_usb.img</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td><code>jinstall</code> image, vRE/VCP (Virtual Control Plane) VM image</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td><code>vFPC</code> image, vFPC/VFP (Virtual Forwarding Plane) VM image</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td><code>hdd</code> image, vRE virtual harddisk image</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
when downloaded from Juniper internal server, the <code>jinstall</code> image (vRE
guest VM) may or may not be included in the VMX tarball. it is available in the
normal Junos releases archives.  tarball downloaded from public URL will always
include <code>jinstall</code> and all other necessary images.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<div class="title">locate the vmx jinstall image</div>
<p>If the tarball does not contain a vRE jinstall image, we can download the image
from other places and copy it over to the same "images" folder after untar the
VMX tarball.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>pings@svl-jtac-tool02:/volume/build/junos/15.1F/daily/20151102.0/ship$
<span class="line-numbers"> 2</span>ls -l | grep install64 | grep vmx
<span class="line-numbers"> 3</span>1 builder   748510583 jinstall64-vmx-15.1F-20151102.0-domestic-signed.tgz
<span class="line-numbers"> 4</span>1 builder          33 jinstall64-vmx-15.1F-20151102.0-domestic-signed.tgz.md5
<span class="line-numbers"> 5</span>1 builder          41 jinstall64-vmx-15.1F-20151102.0-domestic-signed.tgz.sha1
<span class="line-numbers"> 6</span>1 builder  1005715456 jinstall64-vmx-15.1F-20151102.0-domestic.img      #&lt;----
<span class="line-numbers"> 7</span>1 builder          33 jinstall64-vmx-15.1F-20151102.0-domestic.img.md5
<span class="line-numbers"> 8</span>1 builder          41 jinstall64-vmx-15.1F-20151102.0-domestic.img.sha1
<span class="line-numbers"> 9</span>1 builder   748226059 jinstall64-vmx-15.1F-20151102.0-domestic.tgz
<span class="line-numbers">10</span>1 builder          33 jinstall64-vmx-15.1F-20151102.0-domestic.tgz.md5
<span class="line-numbers">11</span>1 builder          41 jinstall64-vmx-15.1F-20151102.0-domestic.tgz.sha1</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
There is no guarantee that an arbitrary combination of <code>jinstall</code> and
<code>vFPC</code> images can work together or not. The publicly released VMX packages
should already include the tested and working combination. Read the offical
instruction and document coming with the software release before starting to
install the VMX.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_prepare_a_work_folder_for_the_installation"><a class="anchor" href="#_prepare_a_work_folder_for_the_installation"></a>1.4. prepare a "work folder" for the installation</h3>
<div class="paragraph">
<p>in my server I organize folder/files in this structure:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>/virtualization             <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 2</span>├── images                  <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 3</span>│   ├── ubuntu.img          <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers"> 4</span>│   ├── vmx_20151102.0      <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers"> 5</span>│   │   ├── build
<span class="line-numbers"> 6</span>│   │   │   └── vmx1
<span class="line-numbers"> 7</span>│   │   │       ├── images
<span class="line-numbers"> 8</span>│   │   │       ├── logs
<span class="line-numbers"> 9</span>│   │   │       │   └── vmx_1448293071.log
<span class="line-numbers">10</span>│   │   │       └── xml
<span class="line-numbers">11</span>│   │   ├── config
<span class="line-numbers">12</span>│   │   │   ├── samples
<span class="line-numbers">13</span>│   │   │   │   ├── vmx.conf.sriov
<span class="line-numbers">14</span>│   │   │   │   ├── vmx.conf.virtio
<span class="line-numbers">15</span>│   │   │   │   └── vmx-galaxy.conf
<span class="line-numbers">16</span>│   │   │   ├── vmx.conf            <i class="conum" data-value="6"></i><b>(6)</b>
<span class="line-numbers">17</span>│   │   │   ├── vmx.conf.sriov1     <i class="conum" data-value="6"></i><b>(6)</b>
<span class="line-numbers">18</span>│   │   │   ├── vmx.conf.sriov2     <i class="conum" data-value="6"></i><b>(6)</b>
<span class="line-numbers">19</span>│   │   │   ├── vmx.conf.ori
<span class="line-numbers">20</span>│   │   │   └── vmx-junosdev.conf
<span class="line-numbers">21</span>│   │   ├── docs
<span class="line-numbers">22</span>
<span class="line-numbers">23</span>...&lt;snipped&gt;...
<span class="line-numbers">24</span>
<span class="line-numbers">25</span>│   └── vmx_20151102.0.tgz  <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">26</span>├── vmx1                    <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">27</span>│   ├── br-ext-generated.xml \
<span class="line-numbers">28</span>│   ├── br-int-generated.xml  |
<span class="line-numbers">29</span>│   ├── cpu_affinitize.sh     |     <i class="conum" data-value="5"></i><b>(5)</b>
<span class="line-numbers">30</span>│   ├── vfconfig-generated.sh |
<span class="line-numbers">31</span>│   ├── vPFE-generated.xml    |
<span class="line-numbers">32</span>│   ├── vRE-generated.xml    /
<span class="line-numbers">33</span>│   └── vmxhdd.img          <i class="conum" data-value="7"></i><b>(7)</b>
<span class="line-numbers">34</span>└── vmx2                    <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">35</span>    ├── br-ext-generated.xml  \
<span class="line-numbers">36</span>    ├── br-int-generated.xml   |
<span class="line-numbers">37</span>    ├── cpu_affinitize.sh      |    <i class="conum" data-value="5"></i><b>(5)</b>
<span class="line-numbers">38</span>    ├── vfconfig-generated.sh  |
<span class="line-numbers">39</span>    ├── vPFE-generated.xml     |
<span class="line-numbers">40</span>    ├── vRE-generated.xml     /
<span class="line-numbers">41</span>    └── vmxhdd.img          <i class="conum" data-value="7"></i><b>(7)</b>
<span class="line-numbers">42</span>
<span class="line-numbers">43</span>27 directories, 136 files</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>parent folder to hold all virtualization files/releases/images</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>"images" sub-folder holds all images for virtualizations</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>VM "images", installation files, tarballs, etc</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>sub-folder to hold all files for installation of multiple VMX VM instances</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>files generated by installation scripts</td>
</tr>
<tr>
<td><i class="conum" data-value="6"></i><b>6</b></td>
<td>VMX installation configuration file</td>
</tr>
<tr>
<td><i class="conum" data-value="7"></i><b>7</b></td>
<td>"hard disk" image file, holding all current VMX guest VM configs</td>
</tr>
</table>
</div>
<div style="page-break-after: always;"></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_install_vmx_using_installation_script_sr_iov"><a class="anchor" href="#_install_vmx_using_installation_script_sr_iov"></a>2. install VMX using installation script (SR-IOV)</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Included in the tarball there is an orchestration script to automate the VMX
setup in the server. running this script without any parameter will provide a
quick help of the usage.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh
<span class="line-numbers"> 2</span>
<span class="line-numbers"> 3</span>Usage: vmx.sh [CONTROL OPTIONS]
<span class="line-numbers"> 4</span>       vmx.sh [LOGGING OPTIONS] [CONTROL OPTIONS]
<span class="line-numbers"> 5</span>       vmx.sh [JUNOS-DEV BIND OPTIONS]
<span class="line-numbers"> 6</span>       vmx.sh [CONSOLE LOGIN OPTIONS]
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>    CONTROL OPTIONS:
<span class="line-numbers"> 9</span>       --install                      : Install And Start vMX
<span class="line-numbers">10</span>       --start                        : Start vMX
<span class="line-numbers">11</span>       --stop                         : Stop vMX
<span class="line-numbers">12</span>       --restart                      : Restart vMX
<span class="line-numbers">13</span>       --status                       : Check Status Of vMX
<span class="line-numbers">14</span>       --cleanup                      : Stop vMX And Cleanup Build Files
<span class="line-numbers">15</span>       --cfg &lt;file&gt;                   : Override With The Specified vmx.conf File
<span class="line-numbers">16</span>       --env &lt;file&gt;                   : Override With The Specified Environment .env File
<span class="line-numbers">17</span>       --build &lt;directory&gt;            : Override With The Specified Directory for Temporary Files
<span class="line-numbers">18</span>       --help                         : This Menu
<span class="line-numbers">19</span>
<span class="line-numbers">20</span>    LOGGING OPTIONS:
<span class="line-numbers">21</span>       -l                             : Enable Logging
<span class="line-numbers">22</span>       -lv                            : Enable Verbose Logging
<span class="line-numbers">23</span>       -lvf                           : Enable Foreground Verbose Logging
<span class="line-numbers">24</span>
<span class="line-numbers">25</span>    JUNOS-DEV BIND OPTIONS:
<span class="line-numbers">26</span>       --bind-dev                     : Bind Junos Devices
<span class="line-numbers">27</span>       --unbind-dev                   : Unbind Junos Devices
<span class="line-numbers">28</span>       --bind-check                   : Check Junos Device Bindings
<span class="line-numbers">29</span>       --cfg &lt;file&gt;                   : Override With The Specified vmx-junosdev.conf File
<span class="line-numbers">30</span>
<span class="line-numbers">31</span>    CONSOLE LOGIN OPTIONS:
<span class="line-numbers">32</span>       --console [vcp|vfp] [vmx_id]   : Login to the Console of VCP/VFP
<span class="line-numbers">33</span>
<span class="line-numbers">34</span>    VFP Image OPTIONS:
<span class="line-numbers">35</span>       --vfp-info &lt;VFP Image Path&gt;    : Display Information About The Specified vFP image
<span class="line-numbers">36</span>
<span class="line-numbers">37</span>Copyright(c) Juniper Networks, 2015</code></pre>
</div>
</div>
<div class="sect2">
<h3 id="_the_vmx_conf_file"><a class="anchor" href="#_the_vmx_conf_file"></a>2.1. the vmx.conf file</h3>
<div class="paragraph">
<p>The config file <code>vmx.conf</code> is a centralized place where all of the installation
parameters and options are defined. The installation script <code>vmx.sh</code> scan this
file as input and generate XML files and shell scripts as output.  the
generated XML files will be read by libvirt/virsh tool later as input
information to setup and manipulate the VMs.</p>
</div>
<div class="paragraph">
<p>The generated shell script will be executed to configure:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>vcpu pinning (both SRIOV and VIRTIO)</p>
</li>
<li>
<p>VF properties (SRIOV only)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For better readability <code>vmx.conf</code> is implemented in YAML format. However, in
VMX the guest VM instances are manipulated via libvirt, which currently solely
relies on XML. This is why there are 2 software modules in the pre-installed
packages to support YAML to XML conversion:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>libyaml-dev  Fast YAML 1.1 parser and emitter library (development)</p>
</li>
<li>
<p>python-yaml  YAML parser and emitter for Python</p>
</li>
</ul>
</div>
<div class="sidebarblock">
<div class="content">
<div class="title">a short introduction about YAML</div>
<div class="paragraph">
<p>YAML (YAML Ain&#8217;t Markup) is a human friendly data serialization language.  VMX
config file uses YAML because it&#8217;s as close to plain English as data
serialization and configuration formats get. The advantage of YAML is that it
does not require curly braces, allowing you to omit quotation marks for strings
in most cases, <strong>relying on indentation</strong> for structure, which makes it much more
readable compared to XML.</p>
</div>
</div>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<i class="fa icon-important" title="Important"></i>
</td>
<td class="content">
<div class="paragraph">
<p>Important tips about YAML:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>indentation matters: YAML relies on indentation to understand the data structure,</p>
</li>
<li>
<p>use <code>space</code> instead of <code>tabs</code>, <code>tabs</code> are not universally supported across
implementations</p>
</li>
<li>
<p>case sensitive</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In short, every space/indentation matters. Taking cautions when modifying the
<code>vmx.conf</code> file. A good practice is to just replace the parameters (numbers,
image path, interface names, MAC, etc) and leave everything else intact.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p><code>vmx.conf</code> template coming with the VMX tarball looks :files</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0/config$ cat vmx.conf</span>
<span class="line-numbers"> 2</span><span class="comment">##############################################################</span>
<span class="line-numbers"> 3</span><span class="comment">#</span>
<span class="line-numbers"> 4</span><span class="comment">#  vmx.conf</span>
<span class="line-numbers"> 5</span><span class="comment">#  Config file for vmx on the hypervisor.</span>
<span class="line-numbers"> 6</span><span class="comment">#  Uses YAML syntax.</span>
<span class="line-numbers"> 7</span><span class="comment">#  Leave a space after &quot;:&quot; to specify the parameter value.</span>
<span class="line-numbers"> 8</span><span class="comment">#</span>
<span class="line-numbers"> 9</span><span class="comment">##############################################################</span>
<span class="line-numbers">10</span>
<span class="line-numbers">11</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">12</span><span class="comment">#Configuration on the host side - management interface, VM images etc.</span>
<span class="line-numbers">13</span><span class="key">HOST</span>:                                                                      <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">14</span>    <span class="key">identifier</span>                : <span class="string"><span class="content">vmx1</span></span>   <span class="comment"># Maximum 4 characters              </span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">15</span>    <span class="key">host-management-interface</span> : <span class="string"><span class="content">eth0                                       </span></span><i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">16</span>    <span class="key">routing-engine-image</span>      : <span class="string"><span class="delimiter">&quot;</span><span class="content">/home/vmx/vmxlite/images/jinstall64-vmx.img</span><span class="delimiter">&quot;</span></span><i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">17</span>    <span class="key">routing-engine-hdd</span>        : <span class="string"><span class="delimiter">&quot;</span><span class="content">/home/vmx/vmxlite/images/vmxhdd.img</span><span class="delimiter">&quot;</span></span>      <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">18</span>    <span class="key">forwarding-engine-image</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">/home/vmx/vmxlite/images/vPFE.img</span><span class="delimiter">&quot;</span></span>        <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">19</span>
<span class="line-numbers">20</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">21</span><span class="comment">#External bridge configuration                                             </span><i class="conum" data-value="5"></i><b>(5)</b>
<span class="line-numbers">22</span><span class="key">BRIDGES</span>:
<span class="line-numbers">23</span>    - <span class="string"><span class="content">type  : external</span></span>
<span class="line-numbers">24</span>      <span class="key">name</span>  : <span class="string"><span class="content">br-ext</span></span>                  <span class="comment"># Max 10 characters</span>
<span class="line-numbers">25</span>
<span class="line-numbers">26</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">27</span><span class="comment">#vRE VM parameters                                                         </span><i class="conum" data-value="6"></i><b>(6)</b>
<span class="line-numbers">28</span><span class="key">CONTROL_PLANE</span>:
<span class="line-numbers">29</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">1</span></span>
<span class="line-numbers">30</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">1024</span></span>
<span class="line-numbers">31</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8601</span></span>
<span class="line-numbers">32</span>
<span class="line-numbers">33</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">34</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">35</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.102.144.94</span></span>
<span class="line-numbers">36</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">0A:00:DD:C0:DE:0E</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">37</span>
<span class="line-numbers">38</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">39</span><span class="comment">#vPFE VM parameters                                                        </span><i class="conum" data-value="7"></i><b>(7)</b>
<span class="line-numbers">40</span><span class="key">FORWARDING_PLANE</span>:
<span class="line-numbers">41</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">6144</span></span>
<span class="line-numbers">42</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">3</span></span>
<span class="line-numbers">43</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8602</span></span>
<span class="line-numbers">44</span>    <span class="key">device-type</span> : <span class="string"><span class="content">virtio                                                   </span></span><i class="conum" data-value="8"></i><b>(8)</b>
<span class="line-numbers">45</span>
<span class="line-numbers">46</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">47</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">48</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.102.144.98</span></span>
<span class="line-numbers">49</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">0A:00:DD:C0:DE:10</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">50</span>
<span class="line-numbers">51</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">52</span><span class="comment">#Interfaces                                                                </span><i class="conum" data-value="9"></i><b>(9)</b>
<span class="line-numbers">53</span><span class="key">JUNOS_DEVICES</span>:
<span class="line-numbers">54</span>   - <span class="string"><span class="content">interface            : ge-0/0/0</span></span>
<span class="line-numbers">55</span>     <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:06:0A:0E:FF:F0</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">56</span>     <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 interface</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">57</span>
<span class="line-numbers">58</span>   - <span class="string"><span class="content">interface            : ge-0/0/1</span></span>
<span class="line-numbers">59</span>     <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:06:0A:0E:FF:F1</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">60</span>     <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 interface</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">61</span>
<span class="line-numbers">62</span>   - <span class="string"><span class="content">interface            : ge-0/0/2</span></span>
<span class="line-numbers">63</span>     <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:06:0A:0E:FF:F2</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">64</span>     <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 interface</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">65</span>
<span class="line-numbers">66</span>   - <span class="string"><span class="content">interface            : ge-0/0/3</span></span>
<span class="line-numbers">67</span>     <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:06:0A:0E:FF:F3</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">68</span>     <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 interface</span><span class="delimiter">&quot;</span></span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>"HOST" config section.</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>ID of the vmx instance. this string will be encoded into the final VM name:
vcp-&lt;ID&gt; or vfp-&lt;ID&gt; .</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>current management interface of the server. the installation script will "move"
the IP/MAC property from this port to an external bridge named "br-ext".</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>vRE/vFPC/Harddisk VM images location.</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>external bridge configuration section: a bridge utility, named <code>br-ext</code>,
will be created, for managment connection from/to the external networks.</td>
</tr>
<tr>
<td><i class="conum" data-value="6"></i><b>6</b></td>
<td>vRE configuration section: this template uses 1 vCPU, 1G mem, console port
8601 to start vRE guest VM. the VM mgmt interface&#8217;s "peer interface"
<sup class="footnote" id="_footnote_peer interface">[<a id="_footnoteref_1" class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</sup>from the host - <code>vcp_ext-vmx1</code> ("attached" to <code>fxp0</code> port from inside the
guest VM), will be configured with the specified MAC address. the corresponding
<code>fxp0</code> interface from inside of vRE VM will inherit same MAC from it.
<sup class="footnote">[<a id="_footnoteref_2" class="footnote" href="#_footnote_2" title="View footnote.">2</a>]</sup></td>
</tr>
<tr>
<td><i class="conum" data-value="7"></i><b>7</b></td>
<td>vFPC configuration section: this template uses 3 vCPU, 6G mem, console port
8602 to build vFPC guest VM. the VM mgmt interface&#8217;s "peer interface"
<sup class="footnoteref">[<a class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</sup> from the host - <code>vfp_ext-vmx1</code> ("attached" to
<code>ext</code> port from side the guest VM) will be configured with the specified MAC
address. the <code>ext</code> interface from inside the vFPC guest VM will inherit same
MAC from it.</td>
</tr>
<tr>
<td><i class="conum" data-value="8"></i><b>8</b></td>
<td>as a KVM implementation, VMX currently supports two type of network IO
virtualization : <code>VT-d</code> + <code>SRIOV</code>, or <code>VIRTIO</code>. this config knob <code>device-type</code>
will determine which IO virtualization technology will be used to build VMX.
This template uses "virtio" IO virtualization.</td>
</tr>
<tr>
<td><i class="conum" data-value="9"></i><b>9</b></td>
<td>VMX router interface configuration section: This is where  the router
<code>ge-0/0/z</code> properties can be configured. depending on <code>device-type</code> value the
available configurable properties will be different. Since this template uses
"virtio" virtualization, only "mac-address" is configurable. more details will
be covered in later sections of this doc.</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_assigning_mac_address"><a class="anchor" href="#_assigning_mac_address"></a>2.2. assigning MAC address</h3>
<div class="paragraph">
<p>the "virtual" MX will use virual NIC running inside the guest VM. so unlike any
real NIC coming with a built-in MAC address which is globally uniquely
assigned, the MAC address of "vNIC" is what you assigned before or after the
VMX instances were brought up.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12179392/86ccb340-b544-11e5-9739-c926e0196ea9.png" alt="86ccb340 b544 11e5 9739 c926e0196ea9">
</div>
<div class="title">Figure 1. internal and external interfaces of VFP and VCP VM</div>
</div>
<div class="paragraph">
<p>To avoid confliction to other external devices,
at least these below MAC addresses needs to be unique in the diagram:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>MAC for ge-0/0/z</p>
</li>
<li>
<p>fxp0 on VRE(VCP)</p>
</li>
<li>
<p>eth1 or ext on VPFE(VFP)</p>
</li>
</ol>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>VFP VM interface name changed in new VMX release.</p>
</div>
<table class="tableblock frame-all grid-all" style="width: 80%;">
<caption class="title">Table 2. VFP interface name</caption>
<colgroup>
<col style="width: 33%;">
<col style="width: 33%;">
<col style="width: 33%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">before 15.1</th>
<th class="tableblock halign-left valign-top">from 15.1</th>
<th class="tableblock halign-left valign-top">roll</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">eth1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ext</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VFP interface facing external networks</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">eth0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">int</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VFP interface for internal use</p></td>
</tr>
</tbody>
</table>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>These MAC addresses will exit the server and be learnt by the external device.</p>
</div>
<div class="paragraph">
<p>Below are internal / isolated interfaces which never communicate with external
devices, MAC address doesn&#8217;t matter for these interfaces:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>em1 on VRE</p>
</li>
<li>
<p>eth0 or int on VPFE</p>
</li>
</ol>
</div>
<div class="paragraph">
<p>"Locally Administered MAC Address" is good candidate to be used in lab test
environment:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>x2-xx-xx-xx-xx-xx
x6-xx-xx-xx-xx-xx
xA-xx-xx-xx-xx-xx
xE-xx-xx-xx-xx-xx</pre>
</div>
</div>
<div class="paragraph">
<p>In my setup I follow this simply rule below to avoid MAC address confliction
with other systems in the lab network:</p>
</div>
<div id="MAC-assignment" class="literalblock">
<div class="content">
<pre>mac-address          : "02:04:17:01:02:02"
                         - ----- -- -- --
                         |    |   |  |  |
                         |    |   |  | <i class="conum" data-value="5"></i><b>(5)</b>
                         |    |   | <i class="conum" data-value="4"></i><b>(4)</b>
                         |    |  <i class="conum" data-value="3"></i><b>(3)</b>
                         |   <i class="conum" data-value="2"></i><b>(2)</b>
                        <i class="conum" data-value="1"></i><b>(1)</b></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>locally administered MAC address</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>last 4 digits of IP (e.g. x.y.4.17) or MAC of management interface.</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>VMX instance number, first VMX instance uses <code>01</code>, second uses <code>02</code>, etc.</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td><code>01</code> for control plane interface (<code>fxp0</code> for VCP, <code>ext</code> for VFP) of a VM,
<code>02</code> for forwarding plane interface (ge-0/0/z) of a VM</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>assign a unique number for each type of interface</td>
</tr>
</table>
</div>
<table class="tableblock frame-all grid-all spread">
<caption class="title">Table 3. an example of MAC address allocation with this rule</caption>
<colgroup>
<col style="width: 33%;">
<col style="width: 33%;">
<col style="width: 33%;">
</colgroup>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">VMX instance1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">vRE  fxp0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:01:01:01</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">vFPC ext</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:01:01:02</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:01:02:01</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:01:02:02</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">VMX instance2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">vRE  fxp0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:02:01:01</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">vFPC ext</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:02:01:02</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:02:02:01</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:04:17:02:02:02</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect2">
<h3 id="_modify_the_vmx_conf_sr_iov"><a class="anchor" href="#_modify_the_vmx_conf_sr_iov"></a>2.3. modify the vmx.conf (SR-IOV)</h3>
<div class="paragraph">
<p>The config file needs to be changed according to the installation plan. this is
vmx.conf file that I use to setup SR-IOV based VMX.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0/config$ cat vmx.conf</span>
<span class="line-numbers"> 2</span><span class="comment">##############################################################</span>
<span class="line-numbers"> 3</span><span class="comment">#</span>
<span class="line-numbers"> 4</span><span class="comment">#  vmx.conf</span>
<span class="line-numbers"> 5</span><span class="comment">#  Config file for vmx on the hypervisor.</span>
<span class="line-numbers"> 6</span><span class="comment">#  Uses YAML syntax.</span>
<span class="line-numbers"> 7</span><span class="comment">#  Leave a space after &quot;:&quot; to specify the parameter value.</span>
<span class="line-numbers"> 8</span><span class="comment">#</span>
<span class="line-numbers"> 9</span><span class="comment">##############################################################</span>
<span class="line-numbers">10</span>
<span class="line-numbers">11</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">12</span><span class="comment">#Configuration on the host side - management interface, VM images etc.</span>
<span class="line-numbers">13</span><span class="key">HOST</span>:
<span class="line-numbers">14</span>    <span class="key">identifier</span>                : <span class="string"><span class="content">vmx1</span></span>   <span class="comment"># Maximum 4 characters</span>
<span class="line-numbers">15</span>    <span class="key">host-management-interface</span> : <span class="string"><span class="content">em1</span></span>
<span class="line-numbers">16</span>    <span class="key">routing-engine-image</span>      : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/jinstall64-vmx-15.1F-20151104.0-domestic.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">17</span>    <span class="key">routing-engine-hdd</span>        : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/vmx1/vmxhdd.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">18</span>    <span class="key">forwarding-engine-image</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/vFPC-20151102.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">19</span>
<span class="line-numbers">20</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">21</span><span class="comment">#External bridge configuration</span>
<span class="line-numbers">22</span><span class="key">BRIDGES</span>:
<span class="line-numbers">23</span>    - <span class="string"><span class="content">type  : external</span></span>
<span class="line-numbers">24</span>      <span class="key">name</span>  : <span class="string"><span class="content">br-ext</span></span>                  <span class="comment"># Max 10 characters</span>
<span class="line-numbers">25</span>
<span class="line-numbers">26</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">27</span><span class="comment">#vRE VM parameters</span>
<span class="line-numbers">28</span><span class="key">CONTROL_PLANE</span>:
<span class="line-numbers">29</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">1</span></span>
<span class="line-numbers">30</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">2048</span></span>
<span class="line-numbers">31</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8816</span></span>
<span class="line-numbers">32</span>
<span class="line-numbers">33</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">34</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">35</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.105</span></span>
<span class="line-numbers">36</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">37</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">38</span><span class="comment">#vPFE VM parameters</span>
<span class="line-numbers">39</span><span class="key">FORWARDING_PLANE</span>:
<span class="line-numbers">40</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">16384</span></span>
<span class="line-numbers">41</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">4                     </span></span><i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">42</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8817</span></span>
<span class="line-numbers">43</span>    <span class="key">device-type</span> : <span class="string"><span class="content">sriov</span></span>
<span class="line-numbers">44</span>
<span class="line-numbers">45</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">46</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">47</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.106</span></span>
<span class="line-numbers">48</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">49</span>
<span class="line-numbers">50</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">51</span><span class="comment">#Interfaces</span>
<span class="line-numbers">52</span><span class="key">JUNOS_DEVICES</span>:
<span class="line-numbers">53</span>    - <span class="string"><span class="content">interface            : ge-0/0/0</span></span>
<span class="line-numbers">54</span>      <span class="key">port-speed-mbps</span>      : <span class="string"><span class="content">10000      </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">55</span>      <span class="key">nic</span>                  : <span class="string"><span class="content">p3p1       </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">56</span>      <span class="key">mtu</span>                  : <span class="string"><span class="content">2000</span></span>             <span class="comment"># DO NOT EDIT </span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">57</span>      <span class="key">virtual-function</span>     : <span class="string"><span class="content">0          </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">58</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04.17:01:02:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">59</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 connects to eth6</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">60</span>
<span class="line-numbers">61</span>    - <span class="string"><span class="content">interface            : ge-0/0/1</span></span>
<span class="line-numbers">62</span>      <span class="key">port-speed-mbps</span>      : <span class="string"><span class="content">10000      </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">63</span>      <span class="key">nic</span>                  : <span class="string"><span class="content">p2p1       </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">64</span>      <span class="key">mtu</span>                  : <span class="string"><span class="content">2000</span></span>             <span class="comment"># DO NOT EDIT</span>
<span class="line-numbers">65</span>      <span class="key">virtual-function</span>     : <span class="string"><span class="content">0          </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">66</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04.17:01:02:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">67</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/1 connects to eth7</span><span class="delimiter">&quot;</span></span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>assign 4 CPUs to vPFE VM</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>SR-IOV only options</td>
</tr>
</table>
</div>
<div class="ulist">
<div class="title">key parameters in this conf file:</div>
<ul>
<li>
<p>my server&#8217;s mgmt interface name is <code>em1</code></p>
</li>
<li>
<p>vRE and vFPC images location will be just the images folder from the untar.ed
installation package. these images can then be shared by all VMX instances.</p>
</li>
<li>
<p>virtual harddisk image <code>vmxhdd.img</code> will be in a seperate folder created
specifically for current VMX instance</p>
</li>
<li>
<p>use console port 88x6 for vRE guest VM and 88x7 for vFPC guest VM, where x =
instance number. Any other number which is not yet in use is fine.</p>
</li>
<li>
<p>MAC addresses are allocated following <a href="#MAC-assignment">the above mentioned rule</a></p>
</li>
<li>
<p>for SR-IOV virtualization, these link properties need to be configured:</p>
<div class="ulist">
<ul>
<li>
<p>physical NIC name</p>
</li>
<li>
<p>VF number</p>
</li>
<li>
<p>MAC address</p>
</li>
<li>
<p>link speed</p>
</li>
<li>
<p>MTU</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
for virtio virtualization, only MAC address can be defined in the config
file. the MTU can be defined in a seperate file <code>vmx-junosdev.conf</code>.
</td>
</tr>
</table>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<i class="fa icon-important" title="Important"></i>
</td>
<td class="content">
<div class="paragraph">
<p>In this example only 4 CPUs were assigned to vPFE VM, which is less than what
is required for full performance in production environment ("performance mode").</p>
</div>
<div class="paragraph">
<p>In VMX 15.1, recommended number of CPU for "performance mode" are calculated as
following:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>1 for host-if</p>
</li>
<li>
<p>1 for flow-manager</p>
</li>
<li>
<p>1 for vmxt process</p>
</li>
<li>
<p>2 per each "IO thread", 1 for each receiving and 1 for sending</p>
</li>
<li>
<p>remaining for "worker thread"</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>non-performance mode or "lite" mode requires less CPU - minimum 3 CPU is fine
to bring up the VFP.</p>
</div>
<div class="paragraph">
<p>Here in lab environment, for test/study purpose I was able to bring up 2 interfaces
SR-IOV VMX with totally 5 CPUs - 1 for VCP and 4 for VFP.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_run_the_installation_script_sr_iov"><a class="anchor" href="#_run_the_installation_script_sr_iov"></a>2.4. run the installation script: SR-IOV</h3>
<div class="paragraph">
<p>After modification of vmx.conf file, we can run the <code>vmx.sh</code> script to setup
the VMX.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">  1</span>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --install
<span class="line-numbers">  2</span>==================================================
<span class="line-numbers">  3</span>    Welcome to VMX
<span class="line-numbers">  4</span>==================================================
<span class="line-numbers">  5</span>Date..............................................11/24/15 21:18:36
<span class="line-numbers">  6</span>VMX Identifier....................................vmx1
<span class="line-numbers">  7</span>Config file......................................./virtualization/images/vmx_20151102.0/config/vmx.conf
<span class="line-numbers">  8</span>Build Directory.................................../virtualization/images/vmx_20151102.0/build/vmx1
<span class="line-numbers">  9</span>Environment file................................../virtualization/images/vmx_20151102.0/env/ubuntu_sriov.env
<span class="line-numbers"> 10</span>Junos Device Type.................................sriov
<span class="line-numbers"> 11</span>Initialize scripts................................[OK]
<span class="line-numbers"> 12</span>Copy images to build directory....................[OK]
<span class="line-numbers"> 13</span>==================================================
<span class="line-numbers"> 14</span>    VMX Environment Setup Completed
<span class="line-numbers"> 15</span>==================================================
<span class="line-numbers"> 16</span>==================================================
<span class="line-numbers"> 17</span>    VMX Install &amp; Start
<span class="line-numbers"> 18</span>==================================================
<span class="line-numbers"> 19</span>Linux distribution................................ubuntu
<span class="line-numbers"> 20</span>Intel IOMMU status................................[Enabled]
<span class="line-numbers"> 21</span>Verify if GRUB needs reboot.......................[No]
<span class="line-numbers"> 22</span>Installation status of qemu-kvm...................[OK]
<span class="line-numbers"> 23</span>Installation status of libvirt-bin................[OK]
<span class="line-numbers"> 24</span>Installation status of bridge-utils...............[OK]
<span class="line-numbers"> 25</span>Installation status of python.....................[OK]
<span class="line-numbers"> 26</span>Installation status of libyaml-dev................[OK]
<span class="line-numbers"> 27</span>Installation status of python-yaml................[OK]
<span class="line-numbers"> 28</span>Installation status of numactl....................[OK]
<span class="line-numbers"> 29</span>Installation status of libnuma-dev................[OK]
<span class="line-numbers"> 30</span>Installation status of libparted0-dev.............[OK]
<span class="line-numbers"> 31</span>Installation status of libpciaccess-dev...........[OK]
<span class="line-numbers"> 32</span>Installation status of libyajl-dev................[OK]
<span class="line-numbers"> 33</span>Installation status of libxml2-dev................[OK]
<span class="line-numbers"> 34</span>Installation status of libglib2.0-dev.............[OK]
<span class="line-numbers"> 35</span>Installation status of libnl-dev..................[OK]
<span class="line-numbers"> 36</span>Check Kernel version..............................[OK]
<span class="line-numbers"> 37</span>Check Qemu version................................[OK]
<span class="line-numbers"> 38</span>Check libvirt version.............................[OK]
<span class="line-numbers"> 39</span>Check virsh connectivity..........................[OK]
<span class="line-numbers"> 40</span>Check IXGBE drivers...............................[OK]
<span class="line-numbers"> 41</span>==================================================
<span class="line-numbers"> 42</span>    Pre-Install Checks Completed
<span class="line-numbers"> 43</span>==================================================
<span class="line-numbers"> 44</span>Check for VM vcp-vmx1.............................[Running]
<span class="line-numbers"> 45</span>Shutdown vcp-vmx1.................................[OK]
<span class="line-numbers"> 46</span>Check for VM vfp-vmx1.............................[Running]
<span class="line-numbers"> 47</span>Shutdown vfp-vmx1.................................[OK]
<span class="line-numbers"> 48</span>Cleanup VM states.................................[OK]
<span class="line-numbers"> 49</span>Check if bridge br-ext exists.....................[Yes]
<span class="line-numbers"> 50</span>Get Configured Management Interface...............em1
<span class="line-numbers"> 51</span>Find existing management gateway..................br-ext
<span class="line-numbers"> 52</span>Mgmt interface needs reconfiguration..............[Yes]
<span class="line-numbers"> 53</span>Gateway interface needs change....................[Yes]
<span class="line-numbers"> 54</span>Check if br-ext has valid IP address..............[Yes]
<span class="line-numbers"> 55</span>Get Management Address............................10.85.4.17
<span class="line-numbers"> 56</span>Get Management Mask...............................255.255.255.128
<span class="line-numbers"> 57</span>Get Management Gateway............................10.85.4.1
<span class="line-numbers"> 58</span>Del em1 from br-ext...............................[OK]
<span class="line-numbers"> 59</span>Configure em1.....................................[Yes]
<span class="line-numbers"> 60</span>Cleanup VM bridge br-ext..........................[OK]
<span class="line-numbers"> 61</span>Cleanup VM bridge br-int-vmx1.....................[OK]
<span class="line-numbers"> 62</span>Cleanup IXGBE drivers.............................[OK]
<span class="line-numbers"> 63</span>==================================================
<span class="line-numbers"> 64</span>    VMX Stop Completed
<span class="line-numbers"> 65</span>==================================================
<span class="line-numbers"> 66</span>Check VCP image...................................[OK]
<span class="line-numbers"> 67</span>Check VFP image...................................[OK]
<span class="line-numbers"> 68</span>VMX Model.........................................FPC
<span class="line-numbers"> 69</span>Check VCP Config image............................[OK]
<span class="line-numbers"> 70</span>Check management interface........................[OK]
<span class="line-numbers"> 71</span>Check interface p3p1..............................[OK]
<span class="line-numbers"> 72</span>Check interface p2p1..............................[OK]
<span class="line-numbers"> 73</span>Setup huge pages to 32768.........................[OK]
<span class="line-numbers"> 74</span>Number of Intel 82599 NICs........................8
<span class="line-numbers"> 75</span>Configuring Intel 82599 Adapters for SRIOV........[OK]
<span class="line-numbers"> 76</span>Number of Virtual Functions created...............[OK]
<span class="line-numbers"> 77</span>Attempt to kill libvirt...........................[OK]
<span class="line-numbers"> 78</span>Attempt to start libvirt..........................[OK]
<span class="line-numbers"> 79</span>Sleep 2 secs......................................[OK]
<span class="line-numbers"> 80</span>Check libvirt support for hugepages...............[OK]
<span class="line-numbers"> 81</span>==================================================
<span class="line-numbers"> 82</span>    System Setup Completed
<span class="line-numbers"> 83</span>==================================================
<span class="line-numbers"> 84</span>Get Management Address of em1.....................[OK]
<span class="line-numbers"> 85</span>Generate libvirt files............................[OK]
<span class="line-numbers"> 86</span>Sleep 2 secs......................................[OK]
<span class="line-numbers"> 87</span>Configure virtual functions for SRIOV.............[OK]
<span class="line-numbers"> 88</span>Find configured management interface..............em1
<span class="line-numbers"> 89</span>Find existing management gateway..................em1
<span class="line-numbers"> 90</span>Check if em1 is already enslaved to br-ext........[No]
<span class="line-numbers"> 91</span>Gateway interface needs change....................[Yes]
<span class="line-numbers"> 92</span>Create br-ext.....................................[OK]
<span class="line-numbers"> 93</span>Get Management Gateway............................10.85.4.1
<span class="line-numbers"> 94</span>Flush em1.........................................[OK]
<span class="line-numbers"> 95</span>Start br-ext......................................[OK]
<span class="line-numbers"> 96</span>Bind em1 to br-ext................................[OK]
<span class="line-numbers"> 97</span>Get Management MAC................................38:ea:a7:37:7c:54
<span class="line-numbers"> 98</span>Assign Management MAC 38:ea:a7:37:7c:54...........[OK]
<span class="line-numbers"> 99</span>Add default gw 10.85.4.1..........................[OK]
<span class="line-numbers">100</span>Create br-int-vmx1................................[OK]
<span class="line-numbers">101</span>Start br-int-vmx1.................................[OK]
<span class="line-numbers">102</span>Check and start default bridge....................[OK]
<span class="line-numbers">103</span>Define vcp-vmx1...................................[OK]
<span class="line-numbers">104</span>Define vfp-vmx1...................................[OK]
<span class="line-numbers">105</span>Wait 2 secs.......................................[OK]
<span class="line-numbers">106</span>Start vcp-vmx1....................................[OK]
<span class="line-numbers">107</span>Start vfp-vmx1....................................[OK]
<span class="line-numbers">108</span>Wait 2 secs.......................................[OK]
<span class="line-numbers">109</span>Perform CPU pinning...............................[OK]
<span class="line-numbers">110</span>==================================================
<span class="line-numbers">111</span>    VMX Bringup Completed
<span class="line-numbers">112</span>==================================================
<span class="line-numbers">113</span>Check if br-ext is created........................[Created]
<span class="line-numbers">114</span>Check if br-int-vmx1 is created...................[Created]
<span class="line-numbers">115</span>Check if VM vcp-vmx1 is running...................[Running]
<span class="line-numbers">116</span>Check if VM vfp-vmx1 is running...................[Running]
<span class="line-numbers">117</span>Check if tap interface vcp_ext-vmx1 exists........[OK]
<span class="line-numbers">118</span>Check if tap interface vcp_int-vmx1 exists........[OK]
<span class="line-numbers">119</span>Check if tap interface vfp_ext-vmx1 exists........[OK]
<span class="line-numbers">120</span>Check if tap interface vfp_int-vmx1 exists........[OK]
<span class="line-numbers">121</span>==================================================
<span class="line-numbers">122</span>    VMX Status Verification Completed.
<span class="line-numbers">123</span>==================================================
<span class="line-numbers">124</span>Log file........................................../dev/null
<span class="line-numbers">125</span>==================================================
<span class="line-numbers">126</span>    Thankyou for using VMX
<span class="line-numbers">127</span>==================================================</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_quick_verification"><a class="anchor" href="#_quick_verification"></a>2.5. quick verification</h3>
<div class="paragraph">
<p>After the installation we need to know if the installed VMX is working well.</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>list running VMs: vRE and vFPC</p>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh list
[sudo] password for ping:
 Id    Name                           State
----------------------------------------------------
 2     vcp-vmx1                       running
 3     vfp-vmx1                       running</pre>
</div>
</div>
</li>
<li>
<p>login to vRE</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ telnet localhost 8816
<span class="line-numbers"> 2</span>Trying ::1...
<span class="line-numbers"> 3</span>Trying 127.0.0.1...
<span class="line-numbers"> 4</span>Connected to localhost.
<span class="line-numbers"> 5</span>Escape character is '^]'.
<span class="line-numbers"> 6</span>
<span class="line-numbers"> 7</span>Amnesiac (ttyd0)
<span class="line-numbers"> 8</span>
<span class="line-numbers"> 9</span>login: root
<span class="line-numbers">10</span>root@% cli
<span class="line-numbers">11</span>root&gt;</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>verify if virtual PFE is "online"</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>labroot&gt; show chassis fpc
<span class="line-numbers">2</span>                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
<span class="line-numbers">3</span>Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
<span class="line-numbers">4</span>  0  Online           Testing   3         0        3      3      2      1         6          0
<span class="line-numbers">5</span>
<span class="line-numbers">6</span>labroot&gt; show chassis fpc pic-status
<span class="line-numbers">7</span>Slot 0   Online       Virtual FPC
<span class="line-numbers">8</span>  PIC 0  Online       Virtual</code></pre>
</div>
</div>
<div class="paragraph">
<p>This may take a couple of minutes.</p>
</div>
</div>
</div>
</li>
<li>
<p>login to vPFE</p>
<div class="listingblock">
<div class="content">
<pre>root@trinity:~# telnet localhost 8817
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.


Wind River Linux 6.0.0.12 vfp-vmx1 console

vfp-vmx1 login: pfe
Password:
pfe@vfp-vmx1:~$ cat /etc/issue.net
Wind River Linux 6.0.0.12 %h</pre>
</div>
</div>
</li>
</ol>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>there are two built-in account to login vfp yocto VM:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>pfe</code> with password <code>pfe</code></p>
</li>
<li>
<p>root account is <code>root</code> with password <code>root</code>.</p>
</li>
</ul>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Later we&#8217;ll demonstrate <a href="#routing-test">ping and OSPF</a> neighborship test. A
more intensive verification requires to enable more features and protocols
(OSPF/BGP/multicast/etc), which is beyond the scope of this doc.</p>
</div>
</div>
<div class="sect2">
<h3 id="_uninstall_cleanup_vmx_with_installation_script"><a class="anchor" href="#_uninstall_cleanup_vmx_with_installation_script"></a>2.6. uninstall (cleanup) VMX with installation script</h3>
<div class="paragraph">
<p>To uninstall VMX just run the same script with <code>--cleanup</code> option:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --cleanup
==================================================
    Welcome to VMX
==================================================
Date..............................................11/24/15 21:25:41
VMX Identifier....................................vmx1
Config file......................................./virtualization/images/vmx_20151102.0/config/vmx.conf
Build Directory.................................../virtualization/images/vmx_20151102.0/build/vmx1
Environment file................................../virtualization/images/vmx_20151102.0/env/ubuntu_sriov.env
Junos Device Type.................................sriov
Initialize scripts................................[OK]
==================================================
    VMX Environment Setup Completed
==================================================
==================================================
    VMX Stop &amp; Cleanup
==================================================
Check if vMX is running...........................[Yes]
Check for VM vcp-vmx1.............................[Running]
Shutdown vcp-vmx1.................................[OK]
Check for VM vfp-vmx1.............................[Running]
Shutdown vfp-vmx1.................................[OK]
Cleanup VM states.................................[OK]
Check if bridge br-ext exists.....................[Yes]
Get Configured Management Interface...............em1
Find existing management gateway..................br-ext
Mgmt interface needs reconfiguration..............[Yes]
Gateway interface needs change....................[Yes]
Check if br-ext has valid IP address..............[Yes]
Get Management Address............................10.85.4.17
Get Management Mask...............................255.255.255.128
Get Management Gateway............................10.85.4.1
Del em1 from br-ext...............................[OK]
Configure em1.....................................[Yes]
Cleanup VM bridge br-ext..........................[OK]
Cleanup VM bridge br-int-vmx1.....................[OK]
Cleanup IXGBE drivers.............................[OK]
==================================================
    VMX Stop Completed
==================================================
Cleanup auto-generated files......................[OK]
==================================================
    VMX Cleanup Completed
==================================================
Log file........................................../dev/null
==================================================
    Thankyou for using VMX
==================================================</pre>
</div>
</div>
<div class="paragraph">
<p>If a different vmx config file was specified when installing VMX, specify the
same as parameter of the script to clean up VMX.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --install --cfg config/vmx.conf.sriov.1
ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --cleanup --cfg config/vmx.conf.sriov.1</pre>
</div>
</div>
<div style="page-break-after: always;"></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_install_vmx_using_installation_script_virtio"><a class="anchor" href="#_install_vmx_using_installation_script_virtio"></a>3. install VMX using installation script (VIRTIO)</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_modify_the_vmx_conf_virtio"><a class="anchor" href="#_modify_the_vmx_conf_virtio"></a>3.1. modify the vmx.conf (virtio)</h3>
<div class="paragraph">
<p>Most configuration parameters in <code>vmx.conf</code> config file to setup <code>VIRTIO</code>
version of VMX are the same as the one used to setup <code>SRIOV</code> VMX as shown
above, a few differences are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>for virtio virtualization, only MAC address can be defined in this config file</p>
</li>
<li>
<p>MTU can to be configured in a seperate <code>junosdev.conf</code> config file</p>
</li>
<li>
<p><code>virtio</code> technology generates seperated virtual NICs that are used to build
VMX; all L1 properties remains in the physical NIC. therefore no need to
specify L1 properties like physical NIC, VF, link speed, etc for VIRTIO setup.</p>
</li>
<li>
<p>To communicate with external networks, virtio virtual NIC can be "bound" to
a physical NIC port, or another virtual NIC, using any existing technologies
like linux bridge, OVS, etc.</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0/config$ cat vmx.conf.virtio.1</span>
<span class="line-numbers"> 2</span><span class="comment">##############################################################</span>
<span class="line-numbers"> 3</span><span class="comment">#</span>
<span class="line-numbers"> 4</span><span class="comment">#  vmx.conf</span>
<span class="line-numbers"> 5</span><span class="comment">#  Config file for vmx on the hypervisor.</span>
<span class="line-numbers"> 6</span><span class="comment">#  Uses YAML syntax.</span>
<span class="line-numbers"> 7</span><span class="comment">#  Leave a space after &quot;:&quot; to specify the parameter value.</span>
<span class="line-numbers"> 8</span><span class="comment">#</span>
<span class="line-numbers"> 9</span><span class="comment">##############################################################</span>
<span class="line-numbers">10</span>
<span class="line-numbers">11</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">12</span><span class="comment">#Configuration on the host side - management interface, VM images etc.</span>
<span class="line-numbers">13</span><span class="key">HOST</span>:
<span class="line-numbers">14</span>    <span class="key">identifier</span>                : <span class="string"><span class="content">vmx1</span></span>   <span class="comment"># Maximum 4 characters</span>
<span class="line-numbers">15</span>    <span class="key">host-management-interface</span> : <span class="string"><span class="content">em1</span></span>
<span class="line-numbers">16</span>    <span class="key">routing-engine-image</span>      : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/jinstall64-vmx-15.1F-20151104.0-domestic.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">17</span>    <span class="key">routing-engine-hdd</span>        : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/vmx1/vmxhdd.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">18</span>    <span class="key">forwarding-engine-image</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/vFPC-20151102.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">19</span>
<span class="line-numbers">20</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">21</span><span class="comment">#External bridge configuration</span>
<span class="line-numbers">22</span><span class="key">BRIDGES</span>:
<span class="line-numbers">23</span>    - <span class="string"><span class="content">type  : external</span></span>
<span class="line-numbers">24</span>      <span class="key">name</span>  : <span class="string"><span class="content">br-ext</span></span>                  <span class="comment"># Max 10 characters</span>
<span class="line-numbers">25</span>
<span class="line-numbers">26</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">27</span><span class="comment">#vRE VM parameters</span>
<span class="line-numbers">28</span><span class="key">CONTROL_PLANE</span>:
<span class="line-numbers">29</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">1</span></span>
<span class="line-numbers">30</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">2048</span></span>
<span class="line-numbers">31</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8816</span></span>
<span class="line-numbers">32</span>
<span class="line-numbers">33</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">34</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">35</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.105</span></span>
<span class="line-numbers">36</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">37</span>        <span class="comment">#macaddr   : &quot;0A:00:DD:C0:DE:0E&quot;</span>
<span class="line-numbers">38</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">39</span><span class="comment">#vPFE VM parameters</span>
<span class="line-numbers">40</span><span class="key">FORWARDING_PLANE</span>:
<span class="line-numbers">41</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">4096</span></span>
<span class="line-numbers">42</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">3</span></span>
<span class="line-numbers">43</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8817</span></span>
<span class="line-numbers">44</span>    <span class="key">device-type</span> : <span class="string"><span class="content">virtio</span></span>
<span class="line-numbers">45</span>
<span class="line-numbers">46</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">47</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">48</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.106</span></span>
<span class="line-numbers">49</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">50</span>        <span class="comment">#macaddr   : &quot;0A:00:DD:C0:DE:10&quot;</span>
<span class="line-numbers">51</span>
<span class="line-numbers">52</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">53</span><span class="comment">#Interfaces</span>
<span class="line-numbers">54</span><span class="key">JUNOS_DEVICES</span>:
<span class="line-numbers">55</span>    - <span class="string"><span class="content">interface            : ge-0/0/0</span></span>
<span class="line-numbers">56</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:02:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">57</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 connects to eth6</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">58</span>
<span class="line-numbers">59</span>    - <span class="string"><span class="content">interface            : ge-0/0/1</span></span>
<span class="line-numbers">60</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:02:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">61</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/1 connects to eth7</span><span class="delimiter">&quot;</span></span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_run_the_installation_script_virtio"><a class="anchor" href="#_run_the_installation_script_virtio"></a>3.2. run the installation script: virtio</h3>
<div class="paragraph">
<p>After modification of vmx config file, we can run the same <code>vmx.sh</code> script to
setup the VMX. This time we use another option: <code>--cfg</code>, to tell the script
where the configuration file can be found. This is necessary if we defined a
seperate config file other than the default <code>config/vmx.conf</code>.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">  1</span>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --install --cfg config/vmx.conf.virtio.1
<span class="line-numbers">  2</span>==================================================
<span class="line-numbers">  3</span>    Welcome to VMX
<span class="line-numbers">  4</span>==================================================
<span class="line-numbers">  5</span>Date..............................................11/30/15 21:40:16
<span class="line-numbers">  6</span>VMX Identifier....................................vmx1
<span class="line-numbers">  7</span>Config file.......................................
<span class="line-numbers">  8</span>    /virtualization/images/vmx_20151102.0/config/vmx.conf.virtio.1
<span class="line-numbers">  9</span>Build Directory.................................../virtualization/images/vmx_20151102.0/build/vmx1
<span class="line-numbers"> 10</span>Environment file................................../virtualization/images/vmx_20151102.0/env/ubuntu_virtio.env
<span class="line-numbers"> 11</span>Junos Device Type.................................virtio
<span class="line-numbers"> 12</span>Initialize scripts................................[OK]
<span class="line-numbers"> 13</span>Copy images to build directory....................[OK]
<span class="line-numbers"> 14</span>==================================================
<span class="line-numbers"> 15</span>    VMX Environment Setup Completed
<span class="line-numbers"> 16</span>==================================================
<span class="line-numbers"> 17</span>==================================================
<span class="line-numbers"> 18</span>    VMX Install &amp; Start
<span class="line-numbers"> 19</span>==================================================
<span class="line-numbers"> 20</span>Linux distribution................................ubuntu
<span class="line-numbers"> 21</span>Check GRUB........................................[Disabled]
<span class="line-numbers"> 22</span>Installation status of qemu-kvm...................[OK]
<span class="line-numbers"> 23</span>Installation status of libvirt-bin................[OK]
<span class="line-numbers"> 24</span>Installation status of bridge-utils...............[OK]
<span class="line-numbers"> 25</span>Installation status of python.....................[OK]
<span class="line-numbers"> 26</span>Installation status of libyaml-dev................[OK]
<span class="line-numbers"> 27</span>Installation status of python-yaml................[OK]
<span class="line-numbers"> 28</span>Installation status of numactl....................[OK]
<span class="line-numbers"> 29</span>Installation status of libnuma-dev................[OK]
<span class="line-numbers"> 30</span>Installation status of libparted0-dev.............[OK]
<span class="line-numbers"> 31</span>Installation status of libpciaccess-dev...........[OK]
<span class="line-numbers"> 32</span>Installation status of libyajl-dev................[OK]
<span class="line-numbers"> 33</span>Installation status of libxml2-dev................[OK]
<span class="line-numbers"> 34</span>Installation status of libglib2.0-dev.............[OK]
<span class="line-numbers"> 35</span>Installation status of libnl-dev..................[OK]
<span class="line-numbers"> 36</span>Check Kernel Version..............................[Disabled]
<span class="line-numbers"> 37</span>Check Qemu Version................................[Disabled]
<span class="line-numbers"> 38</span>Check libvirt Version.............................[Disabled]
<span class="line-numbers"> 39</span>Check virsh connectivity..........................[OK]
<span class="line-numbers"> 40</span>IXGBE Enabled.....................................[Disabled]
<span class="line-numbers"> 41</span>==================================================
<span class="line-numbers"> 42</span>    Pre-Install Checks Completed
<span class="line-numbers"> 43</span>==================================================
<span class="line-numbers"> 44</span>Check for VM vcp-vmx1.............................[Not Running]
<span class="line-numbers"> 45</span>Check for VM vfp-vmx1.............................[Not Running]
<span class="line-numbers"> 46</span>Cleanup VM states.................................[OK]
<span class="line-numbers"> 47</span>Check if bridge br-ext exists.....................[No]
<span class="line-numbers"> 48</span>Cleanup VM bridge br-ext..........................[OK]
<span class="line-numbers"> 49</span>Cleanup VM bridge br-int-vmx1.....................[OK]
<span class="line-numbers"> 50</span>==================================================
<span class="line-numbers"> 51</span>    VMX Stop Completed
<span class="line-numbers"> 52</span>==================================================
<span class="line-numbers"> 53</span>Check VCP image...................................[OK]
<span class="line-numbers"> 54</span>Check VFP image...................................[OK]
<span class="line-numbers"> 55</span>VMX Model.........................................FPC
<span class="line-numbers"> 56</span>Check VCP Config image............................[OK]
<span class="line-numbers"> 57</span>Check management interface........................[OK]
<span class="line-numbers"> 58</span>Setup huge pages to 32768.........................[OK]
<span class="line-numbers"> 59</span>Attempt to kill libvirt...........................[OK]
<span class="line-numbers"> 60</span>Attempt to start libvirt..........................[OK]
<span class="line-numbers"> 61</span>Sleep 2 secs......................................[OK]
<span class="line-numbers"> 62</span>Check libvirt support for hugepages...............[OK]
<span class="line-numbers"> 63</span>==================================================
<span class="line-numbers"> 64</span>    System Setup Completed
<span class="line-numbers"> 65</span>==================================================
<span class="line-numbers"> 66</span>Get Management Address of em1.....................[OK]
<span class="line-numbers"> 67</span>Generate libvirt files............................[OK]
<span class="line-numbers"> 68</span>Sleep 2 secs......................................[OK]
<span class="line-numbers"> 69</span>Find configured management interface..............em1
<span class="line-numbers"> 70</span>Find existing management gateway..................em1
<span class="line-numbers"> 71</span>Check if em1 is already enslaved to br-ext........[No]
<span class="line-numbers"> 72</span>Gateway interface needs change....................[Yes]
<span class="line-numbers"> 73</span>Create br-ext.....................................[OK]
<span class="line-numbers"> 74</span>Get Management Gateway............................10.85.4.1
<span class="line-numbers"> 75</span>Flush em1.........................................[OK]
<span class="line-numbers"> 76</span>Start br-ext......................................[OK]
<span class="line-numbers"> 77</span>Bind em1 to br-ext................................[OK]
<span class="line-numbers"> 78</span>Get Management MAC................................38:ea:a7:37:7c:54
<span class="line-numbers"> 79</span>Assign Management MAC 38:ea:a7:37:7c:54...........[OK]
<span class="line-numbers"> 80</span>Add default gw 10.85.4.1..........................[OK]
<span class="line-numbers"> 81</span>Create br-int-vmx1................................[OK]
<span class="line-numbers"> 82</span>Start br-int-vmx1.................................[OK]
<span class="line-numbers"> 83</span>Check and start default bridge....................[OK]
<span class="line-numbers"> 84</span>Define vcp-vmx1...................................[OK]
<span class="line-numbers"> 85</span>Define vfp-vmx1...................................[OK]
<span class="line-numbers"> 86</span>Wait 2 secs.......................................[OK]
<span class="line-numbers"> 87</span>Start vcp-vmx1....................................[OK]
<span class="line-numbers"> 88</span>Start vfp-vmx1....................................[OK]
<span class="line-numbers"> 89</span>Wait 2 secs.......................................[OK]
<span class="line-numbers"> 90</span>==================================================
<span class="line-numbers"> 91</span>    VMX Bringup Completed
<span class="line-numbers"> 92</span>==================================================
<span class="line-numbers"> 93</span>Check if br-ext is created........................[Created]
<span class="line-numbers"> 94</span>Check if br-int-vmx1 is created...................[Created]
<span class="line-numbers"> 95</span>Check if VM vcp-vmx1 is running...................[Running]
<span class="line-numbers"> 96</span>Check if VM vfp-vmx1 is running...................[Running]
<span class="line-numbers"> 97</span>Check if tap interface vcp_ext-vmx1 exists........[OK]
<span class="line-numbers"> 98</span>Check if tap interface vcp_int-vmx1 exists........[OK]
<span class="line-numbers"> 99</span>Check if tap interface vfp_ext-vmx1 exists........[OK]
<span class="line-numbers">100</span>Check if tap interface vfp_int-vmx1 exists........[OK]
<span class="line-numbers">101</span>==================================================
<span class="line-numbers">102</span>    VMX Status Verification Completed.
<span class="line-numbers">103</span>==================================================
<span class="line-numbers">104</span>Log file........................................../dev/null
<span class="line-numbers">105</span>==================================================
<span class="line-numbers">106</span>    Thankyou for using VMX
<span class="line-numbers">107</span>==================================================</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_uninstall_cleanup_vmx_with_installation_script_2"><a class="anchor" href="#_uninstall_cleanup_vmx_with_installation_script_2"></a>3.3. uninstall (cleanup) VMX with installation script</h3>
<div class="paragraph">
<p>To uninstall VMX just run the same script with and <code>--cleanup</code> option. again we
use <code>--cfg</code> option to specify the <code>vmx.conf</code> file which we used to set up the
VMX :</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --cleanup --cfg config/vmx.conf.virtio.1</span>
<span class="line-numbers"> 2</span><span class="error">==================================================</span>
<span class="line-numbers"> 3</span>    <span class="error">Welcome to VMX</span>
<span class="line-numbers"> 4</span><span class="error">==================================================</span>
<span class="line-numbers"> 5</span><span class="error">Date..............................................11/30/15 21:14:08</span>
<span class="line-numbers"> 6</span><span class="error">VMX Identifier....................................vmx1</span>
<span class="line-numbers"> 7</span><span class="error">Config file.......................................</span>
<span class="line-numbers"> 8</span>    <span class="error">/virtualization/images/vmx_20151102.0/config/vmx.conf.virtio.1</span>
<span class="line-numbers"> 9</span><span class="error">Build Directory.................................../virtualization/images/vmx_20151102.0/build/vmx1</span>
<span class="line-numbers">10</span><span class="error">Environment file................................../virtualization/images/vmx_20151102.0/env/ubuntu_virtio.env</span>
<span class="line-numbers">11</span><span class="error">Junos Device Type.................................virtio</span>
<span class="line-numbers">12</span><span class="error">Initialize scripts................................[OK]</span>
<span class="line-numbers">13</span><span class="error">==================================================</span>
<span class="line-numbers">14</span>    <span class="error">VMX Environment Setup Completed</span>
<span class="line-numbers">15</span><span class="error">==================================================</span>
<span class="line-numbers">16</span><span class="error">==================================================</span>
<span class="line-numbers">17</span>    <span class="error">VMX Stop &amp; Cleanup</span>
<span class="line-numbers">18</span><span class="error">==================================================</span>
<span class="line-numbers">19</span><span class="error">Check if vMX is running...........................[Yes]</span>
<span class="line-numbers">20</span><span class="error">Check for VM vcp-vmx1.............................[Running]</span>
<span class="line-numbers">21</span><span class="error">Shutdown vcp-vmx1.................................[OK]</span>
<span class="line-numbers">22</span><span class="error">Check for VM vfp-vmx1.............................[Running]</span>
<span class="line-numbers">23</span><span class="error">Shutdown vfp-vmx1.................................[OK]</span>
<span class="line-numbers">24</span><span class="error">Cleanup VM states.................................[OK]</span>
<span class="line-numbers">25</span><span class="error">Check if bridge br-ext exists.....................[Yes]</span>
<span class="line-numbers">26</span><span class="error">Get Configured Management Interface...............em1</span>
<span class="line-numbers">27</span><span class="error">Find existing management gateway..................br-ext</span>
<span class="line-numbers">28</span><span class="error">Mgmt interface needs reconfiguration..............[Yes]</span>
<span class="line-numbers">29</span><span class="error">Gateway interface needs change....................[Yes]</span>
<span class="line-numbers">30</span><span class="error">Check if br-ext has valid IP address..............[Yes]</span>
<span class="line-numbers">31</span><span class="error">Get Management Address............................10.85.4.17</span>
<span class="line-numbers">32</span><span class="error">Get Management Mask...............................255.255.255.128</span>
<span class="line-numbers">33</span><span class="error">Get Management Gateway............................10.85.4.1</span>
<span class="line-numbers">34</span><span class="error">Del em1 from br-ext...............................[OK]</span>
<span class="line-numbers">35</span><span class="error">Configure em1.....................................[Yes]</span>
<span class="line-numbers">36</span><span class="error">Cleanup VM bridge br-ext..........................[OK]</span>
<span class="line-numbers">37</span><span class="error">Cleanup VM bridge br-int-vmx1.....................[OK]</span>
<span class="line-numbers">38</span><span class="error">==================================================</span>
<span class="line-numbers">39</span>    <span class="error">VMX Stop Completed</span>
<span class="line-numbers">40</span><span class="error">==================================================</span>
<span class="line-numbers">41</span><span class="error">Cleanup auto-generated files......................[OK]</span>
<span class="line-numbers">42</span><span class="error">==================================================</span>
<span class="line-numbers">43</span>    <span class="error">VMX Cleanup Completed</span>
<span class="line-numbers">44</span><span class="error">==================================================</span>
<span class="line-numbers">45</span><span class="error">Log file........................................../dev/null</span>
<span class="line-numbers">46</span><span class="error">==================================================</span>
<span class="line-numbers">47</span>    <span class="error">Thankyou for using VMX</span>
<span class="line-numbers">48</span><span class="error">==================================================</span></code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_setup_multiple_vmx_instances"><a class="anchor" href="#_setup_multiple_vmx_instances"></a>4. setup multiple VMX instances</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Building multiple VMX instances in a server is no much different than building
a one instance VMX - Just running the installation script multiple time with a
different config file each time will be almost sufficient.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
Depending on the different release, The installation script may or may
not has issue to build multiple instances in one server. There are always works
in progress to improve the script (based on the feedbacks). In case it doesn&#8217;t
work well (e.g bailing out in the middle), you can always
<a href="#install-VMX-manually">install VMX manually</a> In this example we use <code>virtio</code>
to demonstrate the process, <code>SRIOV</code> will be exactly the same process and the
only difference is that more physical NIC or VFs are needed.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>In this section we demonstrate how to set up 2 VIRTIO-based VMX instances,
setting up multiple SRIOV-based VMX instances will be a same process.</p>
</div>
<div class="sect2">
<h3 id="_config_file_for_the_first_vmx_instance"><a class="anchor" href="#_config_file_for_the_first_vmx_instance"></a>4.1. config file for the first VMX instance</h3>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0$ cat config/vmx.conf.virtio.1</span>
<span class="line-numbers"> 2</span><span class="comment">##############################################################</span>
<span class="line-numbers"> 3</span><span class="comment">#</span>
<span class="line-numbers"> 4</span><span class="comment">#  vmx.conf</span>
<span class="line-numbers"> 5</span><span class="comment">#  Config file for vmx on the hypervisor.</span>
<span class="line-numbers"> 6</span><span class="comment">#  Uses YAML syntax.</span>
<span class="line-numbers"> 7</span><span class="comment">#  Leave a space after &quot;:&quot; to specify the parameter value.</span>
<span class="line-numbers"> 8</span><span class="comment">#</span>
<span class="line-numbers"> 9</span><span class="comment">##############################################################</span>
<span class="line-numbers">10</span>
<span class="line-numbers">11</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">12</span><span class="comment">#Configuration on the host side - management interface, VM images etc.</span>
<span class="line-numbers">13</span><span class="key">HOST</span>:
<span class="line-numbers">14</span>    <span class="key">identifier</span>                : <span class="string"><span class="content">vmx1</span></span>   <span class="comment"># Maximum 4 characters</span>
<span class="line-numbers">15</span>    <span class="key">host-management-interface</span> : <span class="string"><span class="content">em1</span></span>
<span class="line-numbers">16</span>    <span class="key">routing-engine-image</span>      : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/jinstall64-vmx-15.1F-20151104.0-domestic.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">17</span>    <span class="key">routing-engine-hdd</span>        : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/vmxhdd.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">18</span>    <span class="key">forwarding-engine-image</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/vFPC-20151102.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">19</span>
<span class="line-numbers">20</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">21</span><span class="comment">#External bridge configuration</span>
<span class="line-numbers">22</span><span class="key">BRIDGES</span>:
<span class="line-numbers">23</span>    - <span class="string"><span class="content">type  : external</span></span>
<span class="line-numbers">24</span>      <span class="key">name</span>  : <span class="string"><span class="content">br-ext</span></span>                  <span class="comment"># Max 10 characters</span>
<span class="line-numbers">25</span>
<span class="line-numbers">26</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">27</span><span class="comment">#vRE VM parameters</span>
<span class="line-numbers">28</span><span class="key">CONTROL_PLANE</span>:
<span class="line-numbers">29</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">1</span></span>
<span class="line-numbers">30</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">2048</span></span>
<span class="line-numbers">31</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8816</span></span>
<span class="line-numbers">32</span>
<span class="line-numbers">33</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">34</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">35</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.105</span></span>
<span class="line-numbers">36</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">37</span>        <span class="comment">#macaddr   : &quot;0A:00:DD:C0:DE:0E&quot;</span>
<span class="line-numbers">38</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">39</span><span class="comment">#vPFE VM parameters</span>
<span class="line-numbers">40</span><span class="key">FORWARDING_PLANE</span>:
<span class="line-numbers">41</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">4096</span></span>
<span class="line-numbers">42</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">3</span></span>
<span class="line-numbers">43</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8817</span></span>
<span class="line-numbers">44</span>    <span class="key">device-type</span> : <span class="string"><span class="content">virtio</span></span>
<span class="line-numbers">45</span>
<span class="line-numbers">46</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">47</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">48</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.106</span></span>
<span class="line-numbers">49</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">50</span>        <span class="comment">#macaddr   : &quot;0A:00:DD:C0:DE:10&quot;</span>
<span class="line-numbers">51</span>
<span class="line-numbers">52</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">53</span><span class="comment">#Interfaces</span>
<span class="line-numbers">54</span><span class="key">JUNOS_DEVICES</span>:
<span class="line-numbers">55</span>    - <span class="string"><span class="content">interface            : ge-0/0/0</span></span>
<span class="line-numbers">56</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:02:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">57</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 connects to eth6</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">58</span>
<span class="line-numbers">59</span>    - <span class="string"><span class="content">interface            : ge-0/0/1</span></span>
<span class="line-numbers">60</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:02:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">61</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/1 connects to eth7</span><span class="delimiter">&quot;</span></span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_config_file_for_the_second_vmx_instance"><a class="anchor" href="#_config_file_for_the_second_vmx_instance"></a>4.2. config file for the second VMX instance</h3>
<div class="paragraph">
<p>Following parameters need to be modified before running the script again - this
is to avoid confliction with the first instance.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>HOST</code> section: <code>identifier</code>, the script will create a folder based on this
string, VM "domain name" will also be based on this
value.</p>
</li>
<li>
<p><code>console_port</code> in both <code>CONTROL_PLANE</code> and <code>FORWARDING_PLANE</code> section</p>
</li>
<li>
<p><code>macaddr</code> in both <code>CONTROL_PLANE</code> and <code>FORWARDING_PLANE</code> section: these are
for VM mgmt port</p>
</li>
<li>
<p><code>JUNOS_DEVICES</code> section: settings for
the Junos ge-0/0/x interface <sup class="footnoteref">[<a class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</sup>:</p>
<div class="ulist">
<ul>
<li>
<p>for VIRTIO-based VMX:</p>
<div class="olist arabic">
<ol class="arabic">
<li>
<p><code>interface</code></p>
</li>
<li>
<p><code>mac-address</code>:</p>
</li>
</ol>
</div>
</li>
<li>
<p>for SRIOV-based VMX, these extra settings are also needed to be configured:</p>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>physical NIC name: <code>nic</code></p>
</li>
<li>
<p><code>virtual-function</code></p>
</li>
<li>
<p><code>port-speed-mbps</code></p>
</li>
<li>
<p><code>mtu</code></p>
</li>
</ol>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0$ cat config/vmx.conf.virtio.2</span>
<span class="line-numbers"> 2</span><span class="comment">##############################################################</span>
<span class="line-numbers"> 3</span><span class="comment">#</span>
<span class="line-numbers"> 4</span><span class="comment">#  vmx.conf</span>
<span class="line-numbers"> 5</span><span class="comment">#  Config file for vmx on the hypervisor.</span>
<span class="line-numbers"> 6</span><span class="comment">#  Uses YAML syntax.</span>
<span class="line-numbers"> 7</span><span class="comment">#  Leave a space after &quot;:&quot; to specify the parameter value.</span>
<span class="line-numbers"> 8</span><span class="comment">#</span>
<span class="line-numbers"> 9</span><span class="comment">##############################################################</span>
<span class="line-numbers">10</span>
<span class="line-numbers">11</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">12</span><span class="comment">#Configuration on the host side - management interface, VM images etc.</span>
<span class="line-numbers">13</span><span class="key">HOST</span>:
<span class="line-numbers">14</span>    <span class="key">identifier</span>                : <span class="string"><span class="content">vmx2</span></span>   <span class="comment"># Maximum 4 characters</span>
<span class="line-numbers">15</span>    <span class="key">host-management-interface</span> : <span class="string"><span class="content">em1</span></span>
<span class="line-numbers">16</span>    <span class="key">routing-engine-image</span>      : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/jinstall64-vmx-15.1F-20151104.0-domestic.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">17</span>    <span class="key">routing-engine-hdd</span>        : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/vmxhdd.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">18</span>    <span class="key">forwarding-engine-image</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">/virtualization/images/vmx_20151102.0/images/vFPC-20151102.img</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">19</span>
<span class="line-numbers">20</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">21</span><span class="comment">#External bridge configuration</span>
<span class="line-numbers">22</span><span class="key">BRIDGES</span>:
<span class="line-numbers">23</span>    - <span class="string"><span class="content">type  : external</span></span>
<span class="line-numbers">24</span>      <span class="key">name</span>  : <span class="string"><span class="content">br-ext</span></span>                  <span class="comment"># Max 10 characters</span>
<span class="line-numbers">25</span>
<span class="line-numbers">26</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">27</span><span class="comment">#vRE VM parameters</span>
<span class="line-numbers">28</span><span class="key">CONTROL_PLANE</span>:
<span class="line-numbers">29</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">1</span></span>
<span class="line-numbers">30</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">2048</span></span>
<span class="line-numbers">31</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8826</span></span>
<span class="line-numbers">32</span>
<span class="line-numbers">33</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">34</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">35</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.107</span></span>
<span class="line-numbers">36</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:02:01:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">37</span>        <span class="comment">#macaddr   : &quot;0A:00:DD:C0:DE:0E&quot;</span>
<span class="line-numbers">38</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">39</span><span class="comment">#vPFE VM parameters</span>
<span class="line-numbers">40</span><span class="key">FORWARDING_PLANE</span>:
<span class="line-numbers">41</span>    <span class="key">memory-mb</span>   : <span class="string"><span class="content">4096</span></span>
<span class="line-numbers">42</span>    <span class="key">vcpus</span>       : <span class="string"><span class="content">3</span></span>
<span class="line-numbers">43</span>    <span class="key">console_port</span>: <span class="string"><span class="content">8827</span></span>
<span class="line-numbers">44</span>    <span class="key">device-type</span> : <span class="string"><span class="content">virtio</span></span>
<span class="line-numbers">45</span>
<span class="line-numbers">46</span>    <span class="key">interfaces</span>  :
<span class="line-numbers">47</span>      - <span class="string"><span class="content">type      : static</span></span>
<span class="line-numbers">48</span>        <span class="key">ipaddr</span>    : <span class="string"><span class="content">10.85.4.108</span></span>
<span class="line-numbers">49</span>        <span class="key">macaddr</span>   : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:02:01:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">50</span>        <span class="comment">#macaddr   : &quot;0A:00:DD:C0:DE:10&quot;</span>
<span class="line-numbers">51</span>
<span class="line-numbers">52</span><span class="head"><span class="head">---</span></span>
<span class="line-numbers">53</span><span class="comment">#Interfaces</span>
<span class="line-numbers">54</span><span class="key">JUNOS_DEVICES</span>:
<span class="line-numbers">55</span>    - <span class="string"><span class="content">interface            : ge-0/0/0</span></span>
<span class="line-numbers">56</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:02:02:01</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">57</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/0 connects to eth6</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">58</span>
<span class="line-numbers">59</span>    - <span class="string"><span class="content">interface            : ge-0/0/1</span></span>
<span class="line-numbers">60</span>      <span class="key">mac-address</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:02:02:02</span><span class="delimiter">&quot;</span></span>
<span class="line-numbers">61</span>      <span class="key">description</span>          : <span class="string"><span class="delimiter">&quot;</span><span class="content">ge-0/0/1 connects to eth7</span><span class="delimiter">&quot;</span></span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_run_installation_script_with_code_cfg_code_option"><a class="anchor" href="#_run_installation_script_with_code_cfg_code_option"></a>4.3. run installation script with <code>--cfg</code> option</h3>
<div class="paragraph">
<p>After the two config files are ready, run <code>vmx.sh</code> to bring them up</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./vmx.sh -lvf --install --cfg config/vmx.conf.virtio.1
./vmx.sh -lvf --install --cfg config/vmx.conf.virtio.2</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_verify_the_2_running_vmx"><a class="anchor" href="#_verify_the_2_running_vmx"></a>4.4. verify the 2 running VMX</h3>
<div class="paragraph">
<div class="title">list the 2 running VMX instances:</div>
<p>vmx1 and vmx2, each contains a vcp and a vfp VM.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo virsh list
 Id    Name                           State
----------------------------------------------------
 2     vcp-vmx1                       running
 3     vfp-vmx1                       running
 5     vcp-vmx2                       running
 6     vfp-vmx2                       running</pre>
</div>
</div>
<div class="sect3">
<h4 id="_ifconfig_virtio"><a class="anchor" href="#_ifconfig_virtio"></a>4.4.1. ifconfig (virtio)</h4>
<div class="paragraph">
<p>Here is the dump of all interfaces after two VMXs are up and running:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ ifconfig -a
br-ext    Link encap:Ethernet  HWaddr 38:ea:a7:37:7c:54               \
          inet addr:10.85.4.17  Bcast:10.85.4.127  Mask:255.255.255.128\
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:26432 errors:0 dropped:0 overruns:0 frame:0        |
          TX packets:10158 errors:0 dropped:0 overruns:0 carrier:0      |
          collisions:0 txqueuelen:0                                     |
          RX bytes:2025742 (2.0 MB)  TX bytes:1184591 (1.1 MB)          |
                                                                        |
br-ext-nic Link encap:Ethernet  HWaddr 52:54:00:9f:a0:77                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                       |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0            |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:500                                   |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                        |
                                                                        |<i class="conum" data-value="1"></i><b>(1)</b>
vcp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:01              |
          inet6 addr: fe80::fc04:17ff:fe01:101/64 Scope:Link            |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0            |
          TX packets:19408 errors:0 dropped:0 overruns:0 carrier:0      |
          collisions:0 txqueuelen:500                                   |
          RX bytes:0 (0.0 B)  TX bytes:1826586 (1.8 MB)                 |
                                                                        |
vcp_ext-vmx2 Link encap:Ethernet  HWaddr fe:04:17:02:01:01              |
          inet6 addr: fe80::fc04:17ff:fe02:101/64 Scope:Link            |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0            |
          TX packets:19248 errors:0 dropped:0 overruns:0 carrier:0      |
          collisions:0 txqueuelen:500                                   |
          RX bytes:0 (0.0 B)  TX bytes:1812426 (1.8 MB)                 |
                                                                        |
vfp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:02              |
          inet6 addr: fe80::fc04:17ff:fe01:102/64 Scope:Link            |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:19409 errors:0 dropped:0 overruns:0 carrier:0      |
          collisions:0 txqueuelen:500                                   |
          RX bytes:2840 (2.8 KB)  TX bytes:1827538 (1.8 MB)             |
                                                                        |
vfp_ext-vmx2 Link encap:Ethernet  HWaddr fe:04:17:02:01:02              |
          inet6 addr: fe80::fc04:17ff:fe02:102/64 Scope:Link            |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:118 errors:0 dropped:0 overruns:0 frame:0          |
          TX packets:19129 errors:0 dropped:0 overruns:0 carrier:0      |
          collisions:0 txqueuelen:500                                  /
          RX bytes:38268 (38.2 KB)  TX bytes:1774106 (1.7 MB)         /

br-int-vmx1 Link encap:Ethernet  HWaddr 52:54:00:56:2f:2b             \
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           \
          RX packets:12823 errors:0 dropped:12587 overruns:0 frame:0    |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:0                                     |
          RX bytes:949948 (949.9 KB)  TX bytes:0 (0.0 B)                |
                                                                        |
br-int-vmx1-nic Link encap:Ethernet  HWaddr 52:54:00:56:2f:2b           |
          BROADCAST MULTICAST  MTU:1500  Metric:1                       |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0            |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:500                                   |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                        |
                                                                        | <i class="conum" data-value="2"></i><b>(2)</b>
vcp_int-vmx1 Link encap:Ethernet  HWaddr fe:54:00:d6:6e:85              |
          inet6 addr: fe80::fc54:ff:fed6:6e85/64 Scope:Link             |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:109078 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:112829 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:500                                   |
          RX bytes:6930175 (6.9 MB)  TX bytes:10139042 (10.1 MB)        |
                                                                        |
vfp_int-vmx1 Link encap:Ethernet  HWaddr fe:54:00:f5:4e:ee              |
          inet6 addr: fe80::fc54:ff:fef5:4eee/64 Scope:Link             |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:108951 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:112338 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:500                                  /
          RX bytes:9928502 (9.9 MB)  TX bytes:7099927 (7.0 MB)        /

br-int-vmx2 Link encap:Ethernet  HWaddr 52:54:00:14:94:e5             \
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           \
          RX packets:12638 errors:0 dropped:12463 overruns:0 frame:0    |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:0                                     |
          RX bytes:938940 (938.9 KB)  TX bytes:0 (0.0 B)                |
                                                                        |
br-int-vmx2-nic Link encap:Ethernet  HWaddr 52:54:00:14:94:e5           |
          BROADCAST MULTICAST  MTU:1500  Metric:1                       |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0            |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:500                                   |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                        |<i class="conum" data-value="3"></i><b>(3)</b>
                                                                        |
vcp_int-vmx2 Link encap:Ethernet  HWaddr fe:54:00:46:01:02              |
          inet6 addr: fe80::fc54:ff:fe46:102/64 Scope:Link              |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:107951 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:111724 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:500                                   |
          RX bytes:6866493 (6.8 MB)  TX bytes:10053803 (10.0 MB)        |
                                                                        |
vfp_int-vmx2 Link encap:Ethernet  HWaddr fe:54:00:b9:67:1d              |
          inet6 addr: fe80::fc54:ff:feb9:671d/64 Scope:Link             |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1            |
          RX packets:107878 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:111185 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:500                                  /
          RX bytes:9845011 (9.8 MB)  TX bytes:7034893 (7.0 MB)        /

em1       Link encap:Ethernet  HWaddr 38:ea:a7:37:7c:54              \
          inet6 addr: fe80::3aea:a7ff:fe37:7c54/64 Scope:Link         \
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           |
          RX packets:1642746 errors:0 dropped:606 overruns:0 frame:0   |
          TX packets:15147394 errors:0 dropped:0 overruns:0 carrier:0  |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:139306967 (139.3 MB)  TX bytes:21847662292 (21.8 GB)|
                                                                       |
em2       Link encap:Ethernet  HWaddr 38:ea:a7:37:7c:55                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                      |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                       |
                                                                       |
em9       Link encap:Ethernet  HWaddr 38:ea:a7:37:7b:d0                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                      |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                       |
                                                                       |
em10      Link encap:Ethernet  HWaddr 38:ea:a7:37:7b:d1                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                      | <i class="conum" data-value="4"></i><b>(4)</b>
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                       |
                                                                       |
p2p1      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:a0                |
          inet6 addr: fe80::3aea:a7ff:fe17:65a0/64 Scope:Link          |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1           |
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:300 (300.0 B)  TX bytes:648 (648.0 B)               |
                                                                       |
p2p2      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:a1                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                      |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                       |
                                                                       |
p3p1      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:84                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                      |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                 |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                       |
                                                                       |
p3p2      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:85                |
          BROADCAST MULTICAST  MTU:1500  Metric:1                      |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:1000                                /
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                     /

ge-0.0.0-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:02:01    \
          inet6 addr: fe80::fc04:17ff:fe01:201/64 Scope:Link    \
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     |
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0     |
          TX packets:3263 errors:0 dropped:0 overruns:0 carrier:0|
          collisions:0 txqueuelen:500                            |
          RX bytes:280 (280.0 B)  TX bytes:169990 (169.9 KB)     |
                                                                 |
ge-0.0.0-vmx2 Link encap:Ethernet  HWaddr fe:04:17:02:02:01      |
          inet6 addr: fe80::fc04:17ff:fe02:201/64 Scope:Link     |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     |
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0     |
          TX packets:3238 errors:0 dropped:0 overruns:0 carrier:0\
          collisions:0 txqueuelen:500                             X <i class="conum" data-value="5"></i><b>(5)</b>
          RX bytes:238 (238.0 B)  TX bytes:168680 (168.6 KB)     /
                                                                 |
ge-0.0.1-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:02:02      |
          inet6 addr: fe80::fc04:17ff:fe01:202/64 Scope:Link     |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0     |
          TX packets:3262 errors:0 dropped:0 overruns:0 carrier:0|
          collisions:0 txqueuelen:500                            |
          RX bytes:0 (0.0 B)  TX bytes:169836 (169.8 KB)         |
                                                                 |
ge-0.0.1-vmx2 Link encap:Ethernet  HWaddr fe:04:17:02:02:02      |
          inet6 addr: fe80::fc04:17ff:fe02:202/64 Scope:Link     |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0     |
          TX packets:3236 errors:0 dropped:0 overruns:0 carrier:0|
          collisions:0 txqueuelen:500                            |
          RX bytes:0 (0.0 B)  TX bytes:168484 (168.4 KB)        /

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:424272 errors:0 dropped:0 overruns:0 frame:0
          TX packets:424272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:22393676 (22.3 MB)  TX bytes:22393676 (22.3 MB)

virbr0    Link encap:Ethernet  HWaddr fe:04:17:01:02:01
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:30 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2751 (2.7 KB)  TX bytes:812 (812.0 B)</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>external bridge</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>internal bridge for vmx1</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>internal bridge for vmx2</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>physical interfaces</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>virtio tap interfaces, peer interfaces <sup class="footnoteref">[<a class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</sup> of JUNOS
ge-0/0/x interfaces in VFP VM</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_virtio_bridging_linux_bridge"><a class="anchor" href="#_virtio_bridging_linux_bridge"></a>4.4.2. virtio bridging: linux bridge</h4>
<div class="paragraph">
<p>virtio does not define anything special for interface bridging. Any open
techniques (linux brige, OVS, etc) can be used for this purpose.</p>
</div>
<div class="paragraph">
<p>A separate config file <code>vmx-junosdev.conf</code> is used to automate virtio interface
bridging using native linux bridge utility.</p>
</div>
<div class="paragraph">
<p>in this example, I&#8217;ll define a new bridge named <code>bridge1</code>, and put following
interfaces into that same bridge:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ge-0/0/0 in vmx1</p>
</li>
<li>
<p>ge-0/0/0 in vmx2</p>
</li>
<li>
<p>p2p1</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The <code>vmx-junosdev.conf</code> file:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">ping@trinity:/virtualization/images/vmx_20151102.0$ cat config/vmx-junosdev.conf.1</span>
<span class="line-numbers"> 2</span><span class="comment">##############################################################</span>
<span class="line-numbers"> 3</span><span class="comment">#</span>
<span class="line-numbers"> 4</span><span class="comment">#  vmx-junos-dev.conf</span>
<span class="line-numbers"> 5</span><span class="comment">#  - Config file for junos device bindings.</span>
<span class="line-numbers"> 6</span><span class="comment">#  - Uses YAML syntax.</span>
<span class="line-numbers"> 7</span><span class="comment">#  - Leave a space after &quot;:&quot; to specify the parameter value.</span>
<span class="line-numbers"> 8</span><span class="comment">#  - For physical NIC, set the 'type' as 'host_dev'</span>
<span class="line-numbers"> 9</span><span class="comment">#  - For junos devices, set the 'type' as 'junos_dev' and</span>
<span class="line-numbers">10</span><span class="comment">#    set the mandatory parameter 'vm-name' to the name of</span>
<span class="line-numbers">11</span><span class="comment">#    the vPFE where the device exists</span>
<span class="line-numbers">12</span><span class="comment">#  - For bridge devices, set the 'type' as 'bridge_dev'</span>
<span class="line-numbers">13</span><span class="comment">#</span>
<span class="line-numbers">14</span><span class="comment">##############################################################</span>
<span class="line-numbers">15</span><span class="key">interfaces</span> :
<span class="line-numbers">16</span>
<span class="line-numbers">17</span>     - <span class="string"><span class="content">link_name  : vmx_bridge1      \</span></span>
<span class="line-numbers">18</span>       <span class="key">mtu</span>        : <span class="string"><span class="content">1500              |</span></span>
<span class="line-numbers">19</span>       <span class="key">endpoint_1</span> :                   <span class="string"><span class="delimiter">|</span><span class="content"></span></span>
<span class="line-numbers">20</span><span class="string"><span class="content">         - type        : junos_dev    \</span></span>
<span class="line-numbers">21</span><span class="string"><span class="content">           vm_name     : vmx1          X  </span></span><i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">22</span><span class="string"><span class="content">           dev_name    : ge-0/0/0     /</span></span>
<span class="line-numbers">23</span>       <span class="key">endpoint_2</span> :                   <span class="string"><span class="delimiter">|</span><span class="content"></span></span>
<span class="line-numbers">24</span><span class="string"><span class="content">         - type        : bridge_dev   |</span></span>
<span class="line-numbers">25</span><span class="string"><span class="content">           dev_name    : bridge1     /</span></span>
<span class="line-numbers">26</span>
<span class="line-numbers">27</span>     - <span class="string"><span class="content">link_name  : vmx_bridge1      \</span></span>
<span class="line-numbers">28</span>       <span class="key">mtu</span>        : <span class="string"><span class="content">1500              |</span></span>
<span class="line-numbers">29</span>       <span class="key">endpoint_1</span> :                   <span class="string"><span class="delimiter">|</span><span class="content"></span></span>
<span class="line-numbers">30</span><span class="string"><span class="content">         - type        : junos_dev    \</span></span>
<span class="line-numbers">31</span><span class="string"><span class="content">           vm_name     : vmx2          X  </span></span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">32</span><span class="string"><span class="content">           dev_name    : ge-0/0/0     /</span></span>
<span class="line-numbers">33</span>       <span class="key">endpoint_2</span> :                   <span class="string"><span class="delimiter">|</span><span class="content"></span></span>
<span class="line-numbers">34</span><span class="string"><span class="content">         - type        : bridge_dev   |</span></span>
<span class="line-numbers">35</span><span class="string"><span class="content">           dev_name    : bridge1     /</span></span>
<span class="line-numbers">36</span>
<span class="line-numbers">37</span>     - <span class="string"><span class="content">link_name  : vmx_bridge1      \</span></span>
<span class="line-numbers">38</span>       <span class="key">mtu</span>        : <span class="string"><span class="content">1500              |</span></span>
<span class="line-numbers">39</span>       <span class="key">endpoint_1</span> :                   <span class="string"><span class="delimiter">|</span><span class="content"></span></span>
<span class="line-numbers">40</span><span class="string"><span class="content">         - type        : host_dev     \</span></span>
<span class="line-numbers">41</span><span class="string"><span class="content">           dev_name    : p2p1          X  </span></span><i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">42</span>       <span class="key">endpoint_2</span> :                   <span class="string"><span class="content">/</span><span class="content"></span></span>
<span class="line-numbers">43</span><span class="string"><span class="content">         - type        : bridge_dev   |</span></span>
<span class="line-numbers">44</span><span class="string"><span class="content">           dev_name    : bridge1     /</span></span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>put junos interface (type <code>junos_dev</code>) <code>ge-0/0/0</code> of first VMX instance
<code>vmx1</code> ,into linux bridge <code>bridge1</code></td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>put junos interface (type <code>junos_dev</code>) <code>ge-0/0/0</code> of second VMX instance
<code>vmx2</code> ,into same linux bridge <code>bridge1</code></td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>put host server interface (type <code>host_dev</code>) <code>p2p1</code> into same linux bridge
<code>bridge1</code></td>
</tr>
</table>
</div>
<div class="paragraph">
<div class="title">bridging before binding:</div>
<p>before <code>binding</code> interfaces to <code>bridge1</code>, all four virtio tap interfaces were
under <code>virbr0</code> interface, which represents the default libvirt network.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ brctl show

bridge name     bridge id               STP enabled     interfaces

br-ext          8000.38eaa7377c54       yes             br-ext-nic
                                                        em1
                                                        vcp_ext-vmx1
                                                        vcp_ext-vmx2
                                                        vfp_ext-vmx1
                                                        vfp_ext-vmx2

br-int-vmx1             8000.525400562f2b       yes     br-int-vmx1-nic
                                                        vcp_int-vmx1
                                                        vfp_int-vmx1

br-int-vmx2             8000.5254001494e5       yes     br-int-vmx2-nic
                                                        vcp_int-vmx2
                                                        vfp_int-vmx2

virbr0          8000.fe0417010201       yes             ge-0.0.0-vmx1
                                                        ge-0.0.0-vmx2
                                                        ge-0.0.1-vmx1
                                                        ge-0.0.1-vmx2</pre>
</div>
</div>
<div class="paragraph">
<div class="title">the bridges diagram</div>
<p>The current bridging structure in this server is illustrated below:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> vcp-vmx1                   br-int-vmx1           br-int-vmx2                   vcp-vmx2
+--------+                         |                 |                          +--------+
|        |                         |                 |                          |        |
|     em1+-------------------------+                 +--------------------------+em1     |
|        |             vcp_int_vmx1|                 |vcp_int_vmx2              |        |
| fxp0   |                         |                 |                          | fxp0   |
+---+----+                         |                 |                          +---+----+
    |               vfp-vmx1       |                 |      vfp-vmx2                |
    |              +--------+      |                 |      +--------+              |
    |              |        | vfp_int-vmx1      vfp-int-vmx2|        |              |
    |              |     int+------+        +-+      +------+int     |              |
    |              |        |               | |             |        |              |
    |              |ext     +---------------+ +-------------+     ext|              |
    |              ++-----+-+ge-0.0.0-vmx1  | |ge-0.0.0-vmx2+-+-----++              |
    |               |     |                 | |               |     |               |
    |               |     +-----------------+ +---------------+     |               |
    |               |   ge-0.0.1-vmx1       | |  ge-0.0.1-vmx2      |               |
    |               |                       +-+                     |               |
    |               |                     virbr0                    |               |
    |               |                                               |               |
    |               |                                               |               |
    |               |                                               |               |
    |vcp_ext-vmx1   |vfp_ext-vmx1                       vfp_ext-vmx2|   vcp_ext-vmx2|
+---+---------------+-----------------------------------------------+---------------+----+
|                                         br-ext    br-ext-nic                           |
+-------------------------------------------------+--------------------------------------+
                                                  |
                                                  |em1</pre>
</div>
</div>
<div class="paragraph">
<p>Now we run the <code>vmx.sh</code> script with <code>--bind-dev</code> option and <code>--cfg</code> option:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ sudo ./vmx.sh --bind-dev --cfg config/vmx-junosdev.conf.1
Checking package ethtool..........................[OK]
Bind Bridge port bridge1(ge-0.0.0-vmx1)...........[OK]
Bind Bridge port bridge1(ge-0.0.0-vmx2)...........[OK]
Bind Bridge port bridge1(p2p1)....................[OK]</pre>
</div>
</div>
<div class="paragraph">
<div class="title">bridging after binding:</div>
<p>After running the <code>vmx.sh</code> script, the peer interface <sup class="footnoteref red" title="Unresolved footnote reference.">[peer
interface]</sup> of JUNOS interface <code>ge-0/0/0</code> from each VMX VM: <code>ge-0.0.0-vmx1</code> and
<code>ge-0.0.0-vmx2</code>, along with the physical port <code>p2p1</code>, now is bound to my new
bridge <code>bridge1</code>.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ brctl show
bridge name     bridge id               STP enabled     interfaces
br-ext          8000.38eaa7377c54       yes             br-ext-nic
                                                        em1
                                                        vcp_ext-vmx1
                                                        vcp_ext-vmx2
                                                        vfp_ext-vmx1
                                                        vfp_ext-vmx2

br-int-vmx1             8000.525400562f2b       yes     br-int-vmx1-nic
                                                        vcp_int-vmx1
                                                        vfp_int-vmx1

br-int-vmx2             8000.5254001494e5       yes     br-int-vmx2-nic
                                                        vcp_int-vmx2
                                                        vfp_int-vmx2

bridge1         8000.38eaa71765a0       no              ge-0.0.0-vmx1
                                                        ge-0.0.0-vmx2
                                                        p2p1

virbr0          8000.fe0417010202       yes             ge-0.0.1-vmx1
                                                        ge-0.0.1-vmx2</pre>
</div>
</div>
<div class="paragraph">
<div class="title">the bridges diagram</div>
<p>Now the briding structure is changed, as can be illustrated from below diagram,
the packet coming from external device, will be bridged to the tap interface
<code>ge-0.0.0-vmx1</code> and <code>ge-0.0.0-vmx2</code> via bridge <code>bridge1</code>, and then goes to the
corresponding <code>ge-0/0/0</code> JUNOS interface in the VMX VM.</p>
</div>
<div class="literalblock">
<div class="content">
<pre> vcp-vmx1                   br-int-vmx1        br-int-vmx2                   vcp-vmx2
+--------+                         +-+           +-+                          +--------+
|        |                         | |           | |                          |        |
|     em1+-------------------------+ |           | +--------------------------+em1     |
|        |             vcp_int_vmx1| |           | |vcp_int_vmx2              |        |
| fxp0   |                         | |           | |                          | fxp0   |
+---+----+                         | |           | |                          +---+----+
    |               vfp-vmx1       | |           | |      vfp-vmx2                |
    |              +--------+      | |           | |      +--------+              |
    |              |        | vfp_int-vmx1    vfp-int-vmx2|        |              |
    |              |     int+------+ |           | +------+int     |              |
    |              |        |      | |    virbr0 | |      |        |              |
    |              |ext     |      +-+     +-+   +-+      |     ext|              |
    |              ++-----+-+\ge-0.0.1-vmx1| |           /+-+-+---++              |
    |               |     |   -------------+ +-----------   |     |               |
    |               |     |                | |              |     |               |
    |               |     |ge-0.0.0-vmx1   +-+ ge-0.0.0-vmx2|     |               |
    |               |    ++---------------------------------+-+   |               |
    |               |    |             bridge1                |   |               |
    |               |    +----------------+-------------------+   |               |
    |               |                     |  p2p1                 |               |
    |vcp_ext-vmx1   |vfp_ext-vmx1                     vfp_ext-vmx2|   vcp_ext-vmx2|
+---+---------------+---------------------------------------------+---------------+----+
|                                       br-ext    br-ext-nic                           |
+-----------------------------------------------+--------------------------------------+
                                                |
                                                |em1</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="routing-test"><a class="anchor" href="#routing-test"></a>4.4.3. routing test</h4>
<div class="paragraph">
<p>A quick way to verify the 2 VMX instances and bridging structure between them
is to run OSPF protocol and ping reachability test between them.</p>
</div>
<div class="paragraph">
<p>For that purpose I enabled IP/OSPF in the external layer 3 switch attached to
the p2p1 interface of the server where VMX instances were built.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>.............................................
. HP server                                 .
.                                           .
.                               bridge1     .
.    +--------+                 +--+        .
.    |VMX1    |                 |  |        .
.    |ge-0/0/0+-----------------+  |        .
.    |1.1.1.11|    ge-0.0.0-vmx1|  |        .
.    +--------+                 |  |        .   external router
.                               |  |        .   +-------+
.                               |  |p2p1    .   |       |
.                               |  +--------.---+1.1.1.13
.                               |  |        .   |       |
.                               |  |        .   +-------+
.    +--------+                 |  |        .
.    |VMX2    |    ge-0.0.0-vmx2|  |        .
.    |ge-0/0/0+-----------------+  |        .
.    |1.1.1.12|                 |  |        .
.    +--------+                 +--+        .
.                                           .
.                                           .
.............................................</pre>
</div>
</div>
<div class="paragraph">
<p>In my setup , both ospf adjacency and ping looks good from first VMX instance
vmx1. This indicates both instances are running well.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>[edit]
root@vmx1# run ping 1.1.1.12
PING 1.1.1.12 (1.1.1.12): 56 data bytes
64 bytes from 1.1.1.12: icmp_seq=0 ttl=64 time=3.081 ms
^C
--- 1.1.1.12 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 3.081/3.081/3.081/0.000 ms

[edit]
root@vmx1# run ping 1.1.1.13
PING 1.1.1.13 (1.1.1.13): 56 data bytes
64 bytes from 1.1.1.13: icmp_seq=0 ttl=64 time=16.046 ms

root@vmx1# run show ospf neighbor
Address          Interface              State     ID               Pri  Dead
1.1.1.13         ge-0/0/0.0             Full      30.0.0.1         128    32
1.1.1.12         ge-0/0/0.0             Full      1.1.1.12         128    32

[edit]
root@vmx1# run show route

inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.0/24         *[Direct/0] 00:55:52
                    &gt; via ge-0/0/0.0
1.1.1.11/32        *[Local/0] 00:55:52
                      Local via ge-0/0/0.0
224.0.0.5/32       *[OSPF/10] 00:02:50, metric 1</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="install-VMX-manually"><a class="anchor" href="#install-VMX-manually"></a>5. install VMX manually</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The installation script is handy to set up VMX, because it helps to automate
most of the configuration tasks, which otherwise all need to be done manually.</p>
</div>
<div class="paragraph">
<p>On the other hand, the script does nothing but to prepare the work environment
for the VMX and run some commands to bring up the two VM images. Therefore,
having the script does not stop us from doing all of the installation work
manually. One advantage of doing so is that if for any reason the script bails
out (e.g sometime the server does not meet all of the presumptions or criterias
the script requires in order to run smoothly), we can still fine tune the
system manually and proceed the installation task. But in order to do that, we
need to know what exactly the script does so we can at least use the step as a
good and tested-and-working reference when something went wrong in the middle.</p>
</div>
<div class="paragraph">
<p>Here is the steps the installation script go through:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>prepare the environment before start:</p>
<div class="ulist">
<ul>
<li>
<p>copy all images into a seperate working folder</p>
</li>
<li>
<p>check if all required software packages are installed</p>
</li>
</ul>
</div>
</li>
<li>
<p>if not done yet, re-compile the ixgbe driver kernel module from source code
for SR-IOV</p>
</li>
<li>
<p>check if there are currently running VMX VMs and bridges with the same name,
if yes,destroy and undefine any VMs and internal bridges with same name</p>
</li>
<li>
<p>remove all VFs if existed, by reloading ixgbe kernel module without <code>max_vfs</code>
option</p>
</li>
<li>
<p>configure "hugepage" and libvirt security and prepare for the VT-d/PCI
passthough ("PCI stub")</p>
</li>
<li>
<p>configure VF feature by reloading ixgbe kernel module with <code>max_vfs</code> option</p>
</li>
<li>
<p>restart libvirt service daemon to make sure libvirt security is successfully
enabled</p>
</li>
<li>
<p>run a python script to parse the YAML config file (<code>vmx.conf</code>) and generate
all necessary XML files which will be used later by libvirt (virsh) to bring up
the VMs. The python script also generate some shell scripts (group of some
commands), which can be used to tune the VM properties (vCPU pinning, VF MAC,
etc) later.</p>
</li>
<li>
<p>for SRIOV-based VMX only, run one of the generated shell script
<code>vfconfig-generated.sh</code>, to configure the property of physical port and VFs.
the script also use VT-d/PCI passthough feature to detach VFs from the host.</p>
</li>
<li>
<p>define external / internal bridge</p>
</li>
<li>
<p>define and bring up VMX VMs</p>
</li>
<li>
<p>perform vCPU pinning</p>
</li>
</ol>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
The running log of the installation script, especially with <code>-lvf</code> debug
option, reveals commands and steps to prepare and setup VMX.
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="_script_generated_xml_files"><a class="anchor" href="#_script_generated_xml_files"></a>5.1. script generated XML files</h3>
<div class="paragraph">
<p>the installation script will generate some XML and shell scripts:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#xml-files">all necessary libvirt XML files</a></p>
<div class="paragraph">
<p>this will be used by the virsh tool to bring up the VMX.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0/build/vmx1/xml$ ls -l | grep xml
-rw-r--r-- 1 root root  383 Nov 24 22:23 br-ext-generated.xml
-rw-r--r-- 1 root root   99 Nov 24 22:23 br-int-generated.xml
-rw-r--r-- 1 root root 2749 Nov 24 22:23 vPFE-generated.xml
-rw-r--r-- 1 root root 3560 Nov 24 22:23 vRE-generated.xml</pre>
</div>
</div>
</li>
<li>
<p><a href="#shell-files">shell scripts</a></p>
<div class="paragraph">
<p>these will be executed by the installation script to configure VF properties
and to configure vCPU pinning</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0/build/vmx1/xml$ ls -l | grep sh
-rw-r--r-- 1 root root  226 Nov 24 22:23 cpu_affinitize.sh
-rw-r--r-- 1 root root  712 Nov 24 22:23 vfconfig-generated.sh</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<div class="title">prepare the XML files for libvirt/virsh</div>
<p>The XML files play an crucial role when bringing up VMs using libvirt/virsh
tool - the libvirt tool relies on XML to store structured data.  Numerous API
functions of libvirt (and their implementation in virsh) take XML documents as
their arguments. XML documents are passed to a XML parser that detects
syntax errors and then are processed internaly.</p>
</div>
<div class="paragraph">
<p>We need to have the XML files ready for use before start to install VMX
manually.</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>create all the necessary XML files</p>
<div class="paragraph">
<p>XML files can be generated by any of the below methods:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>modify existing files, which could be generated by VMX installation
script in other setup (recommended)</p>
</li>
<li>
<p>creat them from scratch (not recommeneded)</p>
</li>
</ul>
</div>
</li>
<li>
<p>update all parameters in the XML files</p>
<div class="ulist">
<ul>
<li>
<p>domain/bridge names</p>
</li>
<li>
<p>image path</p>
</li>
<li>
<p>MAC of ext mgmt (vcp_ext-vmx) interface in external bridge</p>
</li>
<li>
<p>console/serial port</p>
</li>
<li>
<p>for VFP VM: hostdev (SR-IOV VF) function# and MAC#</p>
</li>
<li>
<p>external bridge xml: external MAC and IP of mgmt i/f (vcp_ext-vmx)</p>
</li>
</ul>
</div>
</li>
</ol>
</div>
<div class="paragraph">
<p>The XML files were printed in <a href="#appendix">appendix</a> for reference.</p>
</div>
</div>
<div class="sect2">
<h3 id="_manual_installation_steps"><a class="anchor" href="#_manual_installation_steps"></a>5.2. manual installation steps</h3>
<div class="olist arabic">
<ol class="arabic">
<li>
<p><a href="#enable-iommu">enable iommu/VT-d</a> (SR-IOV only)</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p>enabling IOMMU/VT-d allows the physical PCI devices to be able to be assigned
to the VM directly.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
skip if this was done once
</td>
</tr>
</table>
</div>
</div>
</div>
</li>
<li>
<p>recompile and reload ixgbe kernel driver module (SR-IOV only)</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p>Juniper modified IXGBE kernel driver is required for VMX.
the driver needs to be re-compiled from source code that comes with the
installation package.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
skip if this was done once
</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
<div class="paragraph">
<p>to verify if the ixgbe version is right:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ modinfo ixgbe | grep version
version:        3.19.1
srcversion:     B97B1E7CF79A25F5E4D7B96
vermagic:       3.13.0-32-generic SMP mod_unload modversions</pre>
</div>
</div>
</td>
</tr>
</table>
</div>
<div class="ulist">
<ul>
<li>
<p>compile and install ixgbe driver from source code</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre>cd /virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src
rm -f ixgbe.ko
make install
sleep 5
cmp ixgbe.ko /lib/modules/3.13.0-32-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko</pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p>reload ixgbe driver</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre>rmmod ixgbevf
rmmod ixgbe
modprobe ixgbe</pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>setup hugepage</p>
<div class="openblock">
<div class="content">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
skip if this was done once
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>sudo -i
<span class="line-numbers"> 2</span>echo 32768 &gt; /proc/sys/vm/nr_overcommit_hugepages
<span class="line-numbers"> 3</span>exit
<span class="line-numbers"> 4</span>
<span class="line-numbers"> 5</span>mkdir /HugePage_vPFE
<span class="line-numbers"> 6</span>sudo mount -t hugetlbfs hugetlbfs /HugePage_vPFE
<span class="line-numbers"> 7</span>sudo mount | grep &quot;HugePage_vPFE&quot;
<span class="line-numbers"> 8</span>
<span class="line-numbers"> 9</span>sudo service libvirt-bin restart
<span class="line-numbers">10</span>
<span class="line-numbers">11</span>cat /etc/apparmor.d/abstractions/libvirt-qemu  | grep HugePage_vPFE
<span class="line-numbers">12</span>sudo -i
<span class="line-numbers">13</span>echo &quot;owner \&quot;/HugePage_vPFE/libvirt/qemu/**\&quot; rw,&quot; &gt;&gt; /etc/apparmor.d/abstractions/libvirt-qemu
<span class="line-numbers">14</span>exit</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="ulist">
<ul>
<li>
<p>VM won&#8217;t start properly without hugetlbfs mounted beforehand.</p>
<div class="listingblock">
<div class="content">
<pre>ping@matrix:~$ ls /HugePage_vPFE/
ping@matrix:~$ mount | grep "HugePage_vPFE"
ping@matrix:~$
ping@matrix:~$ sudo virsh start AVPN-vfp
error: Failed to start domain AVPN-vfp
error: internal error: hugetlbfs filesystem is not mounted or disabled by administrator config</pre>
</div>
</div>
</li>
<li>
<p>libvirt needs to be restarted after hugetlbfs got mounted:</p>
<div class="listingblock">
<div class="content">
<pre>ping@matrix:~$ sudo mount -t hugetlbfs hugetlbfs /HugePage_vPFE
ping@matrix:~$ sudo virsh start AVPN-vfp
error: Failed to start domain AVPN-vfp
error: internal error: hugetlbfs filesystem is not mounted or disabled by administrator config

ping@matrix:~$ ls /HugePage_vPFE/

ping@matrix:~$ sudo service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 46220

ping@matrix:~$ ls /HugePage_vPFE/
libvirt
ping@matrix:~$ sudo virsh start AVPN-vfp
Domain AVPN-vfp started</pre>
</div>
</div>
</li>
</ul>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
</li>
<li>
<p>prepare for pci-stub (SR-IOV only)</p>
<div class="openblock">
<div class="content">
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
skip if this was done once
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers">1</span>sudo modprobe pci-stub
<span class="line-numbers">2</span>
<span class="line-numbers">3</span>sudo -i
<span class="line-numbers">4</span>echo &quot;8086 10ed&quot; &gt; /sys/bus/pci/drivers/pci-stub/new_id
<span class="line-numbers">5</span>&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">6</span>exit</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>reload ixgbe kernel driver and configure SR-IOV VF (SR-IOV only)</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers">1</span>sudo rmmod ixgbevf; sudo rmmod ixgbe; \
<span class="line-numbers">2</span>sudo modprobe ixgbe max_vfs=2,2,2,2,2,2,2,2; \
<span class="line-numbers">3</span>sudo modprobe tun ; \                              <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">4</span>sleep 5; sudo brctl addif br-ext em9</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td><a href="https://www.kernel.org/doc/Documentation/networking/tuntap.txt">tun/tap</a>
interface: TUN (namely network TUNnel) simulates a network layer device and it
operates with layer 3 packets like IP packets. TAP (namely network tap)
simulates a link layer device and it operates with layer 2 packets like
Ethernet frames. TUN is used with routing, while TAP is used for creating a
network bridge, see <a href="https://en.wikipedia.org/wiki/TUN/TAP">wiki</a>.  In VMX "tap"
interface and bridges will be used and that is why this kernel module is
required. this is not directly related to SR-IOV.</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
these VF configs, and also a lot of the other configuration here won&#8217;t be
persistent across system reboot.
</td>
</tr>
</table>
</div>
</div>
</div>
</li>
<li>
<p>configure interface and VF properties (SR-IOV only)</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p>follow the <a href="#MAC-assignment">MAC assignment</a> rule to assign MAC for the 2 VMX
instance</p>
</div>
<div class="listingblock">
<div class="title">MAC address assignment</div>
<div class="content">
<pre>02:04:17:01:01:01 vcp-vmx1 fxp0
02:04:17:01:01:02 vfp-vmx1 eth0
02:04:17:01:02:01 vmx1 ge-0/0/0 p3p1 vf0
02:04:17:01:02:02 vmx1 ge-0/0/1 p2p1 vf0

02:04:17:02:01:01 vcp-vmx2 fxp0
02:04:17:02:01:02 vfp-vmx2 eth0
02:04:17:02:02:01 vmx2 ge-0/0/0 p3p1 vf1
02:04:17:02:02:02 vmx2 ge-0/0/1 p2p1 vf1</pre>
</div>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers">1</span>export XML_FOLDER=&quot;/virtualization/vmx1/xml&quot;
<span class="line-numbers">2</span>sudo sh $XML_FOLDER/vfconfig-generated.sh
<span class="line-numbers">3</span>export XML_FOLDER=&quot;/virtualization/vmx2/xml&quot;
<span class="line-numbers">4</span>sudo sh $XML_FOLDER/vfconfig-generated.sh</code></pre>
</div>
</div>
<div class="paragraph">
<p>or</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>#VMX1
<span class="line-numbers"> 2</span>sudo ifconfig p3p1 up promisc allmulti mtu 2000
<span class="line-numbers"> 3</span>sudo ifconfig p2p1 up promisc allmulti mtu 2000
<span class="line-numbers"> 4</span>sudo ip link set p3p1 vf 0 mac 02:04:17:01:02:01
<span class="line-numbers"> 5</span>sudo ip link set p2p1 vf 0 mac 02:04:17:01:02:02
<span class="line-numbers"> 6</span>sudo ip link set p3p1 vf 0 rate 10000
<span class="line-numbers"> 7</span>sudo ip link set p2p1 vf 0 rate 10000
<span class="line-numbers"> 8</span>sudo ip link set p3p1 vf 0 spoofchk off
<span class="line-numbers"> 9</span>sudo ip link set p2p1 vf 0 spoofchk off
<span class="line-numbers">10</span>
<span class="line-numbers">11</span>#VMX2
<span class="line-numbers">12</span>sudo ifconfig p3p1 up promisc allmulti mtu 2000
<span class="line-numbers">13</span>sudo ifconfig p2p1 up promisc allmulti mtu 2000
<span class="line-numbers">14</span>sudo ip link set p3p1 vf 1 mac 02:04:17:02:02:01
<span class="line-numbers">15</span>sudo ip link set p2p1 vf 1 mac 02:04:17:02:02:02
<span class="line-numbers">16</span>sudo ip link set p3p1 vf 1 rate 10000
<span class="line-numbers">17</span>sudo ip link set p2p1 vf 1 rate 10000
<span class="line-numbers">18</span>sudo ip link set p3p1 vf 1 spoofchk off
<span class="line-numbers">19</span>sudo ip link set p2p1 vf 1 spoofchk off</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>pci-stub hide (SR-IOV only)</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p>unbind the VF from kernel driver (host) and bind it to pci-stub module,
effectively hide this VF from host kernel and the VF will later be assigned
directly to the VM.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
skip if this was done once
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>sudo -i
<span class="line-numbers"> 2</span>#p3p1 vf0
<span class="line-numbers"> 3</span>echo 0000:23:10.0 &gt; /sys/bus/pci/devices/0000:23:10.0/driver/unbind
<span class="line-numbers"> 4</span>echo 0000:23:10.0 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers"> 5</span>#p3p1 vf1
<span class="line-numbers"> 6</span>echo 0000:23:10.2 &gt; /sys/bus/pci/devices/0000:23:10.2/driver/unbind
<span class="line-numbers"> 7</span>echo 0000:23:10.2 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers"> 8</span>
<span class="line-numbers"> 9</span>#p2p1 vf0
<span class="line-numbers">10</span>echo 0000:06:10.0 &gt; /sys/bus/pci/devices/0000:06:10.0/driver/unbind
<span class="line-numbers">11</span>echo 0000:06:10.0 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">12</span>#p2p1 vf1
<span class="line-numbers">13</span>echo 0000:06:10.2 &gt; /sys/bus/pci/devices/0000:06:10.2/driver/unbind
<span class="line-numbers">14</span>echo 0000:06:10.2 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">15</span>exit</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>config VN &amp; bridges</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>#vmx1:
<span class="line-numbers"> 2</span>export XML_FOLDER=&quot;/virtualization/vmx1&quot;
<span class="line-numbers"> 3</span>sudo virsh net-define $XML_FOLDER/br-ext-generated.xml
<span class="line-numbers"> 4</span>
<span class="line-numbers"> 5</span>sudo ifconfig em1 0; \
<span class="line-numbers"> 6</span>sudo virsh net-start br-ext; \
<span class="line-numbers"> 7</span>sudo brctl addif br-ext em1; \
<span class="line-numbers"> 8</span>sudo route add default gw 10.85.4.1
<span class="line-numbers"> 9</span>sudo ifconfig br-ext hw ether 38:ea:a7:37:7c:54;
<span class="line-numbers">10</span>
<span class="line-numbers">11</span>sudo virsh net-define $XML_FOLDER/br-int-generated.xml
<span class="line-numbers">12</span>sudo virsh net-start br-int-vmx1
<span class="line-numbers">13</span>
<span class="line-numbers">14</span>#vmx2:
<span class="line-numbers">15</span>export XML_FOLDER=&quot;/virtualization/vmx2&quot;
<span class="line-numbers">16</span>
<span class="line-numbers">17</span>sudo virsh net-define $XML_FOLDER/br-int-generated.xml
<span class="line-numbers">18</span>sudo virsh net-start br-int-vmx2</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>define and start VMs using the generated xml file:</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>sudo virsh net-list | grep default
<span class="line-numbers"> 2</span>
<span class="line-numbers"> 3</span>#vmx1
<span class="line-numbers"> 4</span>sudo virsh define /virtualization/vmx1/vRE-generated1.xml
<span class="line-numbers"> 5</span>sudo virsh define /virtualization/vmx1/vPFE-generated1.xml
<span class="line-numbers"> 6</span>sudo virsh start vcp-vmx1
<span class="line-numbers"> 7</span>sudo virsh start vfp-vmx1
<span class="line-numbers"> 8</span>
<span class="line-numbers"> 9</span>#vmx2
<span class="line-numbers">10</span>sudo virsh define /virtualization/vmx2/vRE-generated2.xml
<span class="line-numbers">11</span>sudo virsh define /virtualization/vmx2/vPFE-generated2.xml
<span class="line-numbers">12</span>sudo virsh start vcp-vmx2
<span class="line-numbers">13</span>sudo virsh start vfp-vmx2</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>vcpupin:</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers">1</span>export XML_FOLDER=&quot;/virtualization/vmx1&quot;
<span class="line-numbers">2</span>sudo sh $XML_FOLDER/cpu_affinitize.sh
<span class="line-numbers">3</span>export XML_FOLDER=&quot;/virtualization/vmx2&quot;
<span class="line-numbers">4</span>sudo sh $XML_FOLDER/cpu_affinitize.sh</code></pre>
</div>
</div>
<div class="paragraph">
<p>or</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>#vmx1:
<span class="line-numbers"> 2</span>sudo virsh emulatorpin vcp-vmx1 0
<span class="line-numbers"> 3</span>sudo virsh emulatorpin vfp-vmx1 0
<span class="line-numbers"> 4</span>
<span class="line-numbers"> 5</span>sudo virsh vcpupin vcp-vmx1 0 7
<span class="line-numbers"> 6</span>
<span class="line-numbers"> 7</span>sudo virsh vcpupin vfp-vmx1 0 0
<span class="line-numbers"> 8</span>sudo virsh vcpupin vfp-vmx1 1 1
<span class="line-numbers"> 9</span>sudo virsh vcpupin vfp-vmx1 2 2
<span class="line-numbers">10</span>sudo virsh vcpupin vfp-vmx1 3 3
<span class="line-numbers">11</span>
<span class="line-numbers">12</span>#vmx2:
<span class="line-numbers">13</span>sudo virsh emulatorpin vcp-vmx2 0
<span class="line-numbers">14</span>sudo virsh emulatorpin vfp-vmx2 0
<span class="line-numbers">15</span>
<span class="line-numbers">16</span>sudo virsh vcpupin vcp-vmx2 0 15
<span class="line-numbers">17</span>
<span class="line-numbers">18</span>sudo virsh vcpupin vfp-vmx2 0 8
<span class="line-numbers">19</span>sudo virsh vcpupin vfp-vmx2 1 9
<span class="line-numbers">20</span>sudo virsh vcpupin vfp-vmx2 2 10
<span class="line-numbers">21</span>sudo virsh vcpupin vfp-vmx2 3 11</code></pre>
</div>
</div>
</div>
</div>
</li>
</ol>
</div>
<div class="paragraph">
<div class="title">multiple VMX instances</div>
<p>using the above files generated from the installation script, it&#8217;s easy to
bring up multiple instances of VMX manually.</p>
</div>
</div>
<div class="sect2">
<h3 id="_verify_installations"><a class="anchor" href="#_verify_installations"></a>5.3. verify installations</h3>
<div class="literalblock">
<div class="title">show running VMX instances:</div>
<div class="content">
<pre>ping@trinity:/virtualization/vmx1$ sudo virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     vcp-vmx1                       running
 6     vfp-vmx1                       running
 7     vcp-vmx2                       running
 8     vfp-vmx2                       running</pre>
</div>
</div>
<div class="listingblock">
<div class="title">identify console ports of VCP and VFP VMs for both instance:</div>
<div class="content">
<pre>ping@trinity:/virtualization/vmx1$ netstat -na | grep ":8[12][67] "
tcp        0      0 127.0.0.1:8826          0.0.0.0:*               LISTEN              #&lt;------
tcp        0      0 127.0.0.1:8827          0.0.0.0:*               LISTEN              #&lt;------
tcp        0      0 127.0.0.1:8816          0.0.0.0:*               LISTEN              #&lt;------
tcp        0      0 127.0.0.1:8817          0.0.0.0:*               LISTEN              #&lt;------
tcp        0      0 127.0.0.1:8816          127.0.0.1:42765         ESTABLISHED
tcp        0      0 127.0.0.1:41726         127.0.0.1:8826          ESTABLISHED
tcp        0      0 127.0.0.1:42765         127.0.0.1:8816          ESTABLISHED
tcp        0      0 127.0.0.1:8826          127.0.0.1:41726         ESTABLISHED</pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>the port number that qemu is listening can be configured to any number,
as long as the port is unique in the system wide and not in use by other
applications. in the above example I use 88x6 for VCP console port, and 88x7
for VFP console port, where x is instance number.</p>
</div>
<div class="paragraph">
<p>Another approach, is to use the same port number for all instances, but
changing the local IP to 127.0.0.x where x can be instance number - just make
sure the local "socket" (IP + port pair) is unique will be sufficient.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu4.54:~$ sudo netstat -nap | grep qemu
[sudo] password for ping:
tcp        0      0 127.0.0.2:8896          0.0.0.0:*               LISTEN      33284/qemu-system-x     #&lt;------
tcp        0      0 127.0.0.1:8896          0.0.0.0:*               LISTEN      15030/qemu-system-x     #&lt;------
tcp        0      0 127.0.0.2:8897          0.0.0.0:*               LISTEN      33330/qemu-system-x     #&lt;------
tcp        0      0 127.0.0.1:8897          0.0.0.0:*               LISTEN      15075/qemu-system-x     #&lt;------
tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      15030/qemu-system-x
tcp        0      0 127.0.0.1:5901          0.0.0.0:*               LISTEN      15075/qemu-system-x
tcp        0      0 127.0.0.2:5902          0.0.0.0:*               LISTEN      33284/qemu-system-x
tcp        0      0 127.0.0.2:5903          0.0.0.0:*               LISTEN      33330/qemu-system-x
unix  2      [ ACC ]     STREAM     LISTENING     62021    15030/qemu-system-x //lib/libvirt/qemu/MIS-VMX-VCP.monitor
unix  2      [ ACC ]     STREAM     LISTENING     23706    15075/qemu-system-x //lib/libvirt/qemu/MIS-VMX-VFP.monitor
unix  2      [ ACC ]     STREAM     LISTENING     62277    33284/qemu-system-x //lib/libvirt/qemu/AVPN-VMX-VCP.monitor
unix  2      [ ACC ]     STREAM     LISTENING     24867    33330/qemu-system-x //lib/libvirt/qemu/AVPN-VMX-VFP.monitor
unix  3      [ ]         STREAM     CONNECTED     63371    33330/qemu-system-x //lib/libvirt/qemu/AVPN-VMX-VFP.monitor
unix  3      [ ]         STREAM     CONNECTED     27800    15030/qemu-system-x //lib/libvirt/qemu/MIS-VMX-VCP.monitor
unix  3      [ ]         STREAM     CONNECTED     63366    33284/qemu-system-x //lib/libvirt/qemu/AVPN-VMX-VCP.monitor
unix  3      [ ]         STREAM     CONNECTED     59462    15075/qemu-system-x //lib/libvirt/qemu/MIS-VMX-VFP.monitor</pre>
</div>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<div class="title">login to console of the two VMX instances:</div>
<p>vmx1:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ telnet localhost 8816
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Amnesiac (ttyd0)

login: root

--- JUNOS 15.1F-20151104.0 built 2015-11-04 05:39:37 UTC
root@% cli
root &gt;

root&gt; show version
Model: vmx
Junos: 15.1F-20151104.0
JUNOS Base OS boot [15.1F-20151104.0]
JUNOS Base OS Software Suite [15.1F-20151104.0]
...

labroot&gt; show chassis fpc
                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
  0  Online           Testing   3         0        3      3      2      1         6          0

labroot&gt; show chassis fpc pic-status
Slot 0   Online       Virtual FPC
  PIC 0  Online       Virtual</pre>
</div>
</div>
<div class="paragraph">
<p>vmx2:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ telnet localhost 8826
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
......
Amnesiac (ttyd0)
login: lab
root@% cli
root&gt;</pre>
</div>
</div>
<div class="listingblock">
<div class="title">ping test:</div>
<div class="content">
<pre>labroot@vmx1# run show interfaces ge-0/0/0 terse
Interface               Admin Link Proto    Local                 Remote
ge-0/0/0                up    up
ge-0/0/0.0              up    up   inet     1.1.1.1/24
                                   multiservice

root@vmx2# run show interfaces ge-0/0/0 terse
Interface               Admin Link Proto    Local                 Remote
ge-0/0/0                up    up
ge-0/0/0.0              up    up   inet     1.1.1.2/24
                                   multiservice

[edit]
labroot@vmx1# run ping 1.1.1.2
PING 1.1.1.2 (1.1.1.2): 56 data bytes
64 bytes from 1.1.1.2: icmp_seq=0 ttl=64 time=1.694 ms
64 bytes from 1.1.1.2: icmp_seq=1 ttl=64 time=1.751 ms
64 bytes from 1.1.1.2: icmp_seq=2 ttl=64 time=1.817 ms
^C
--- 1.1.1.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.694/1.754/1.817/0.050 ms

[edit]
labroot@vmx1# run show arp
MAC Address       Address         Name                      Interface               Flags
02:04:17:02:02:01 1.1.1.2         1.1.1.2                   ge-0/0/0.0              none
52:54:00:9c:b3:f2 128.0.0.16      128.0.0.16                em1.0                   none
Total entries: 2

[edit]
labroot@vmx1# run ping 10.85.4.107
PING 10.85.4.107 (10.85.4.107): 56 data bytes
64 bytes from 10.85.4.107: icmp_seq=0 ttl=64 time=0.822 ms
64 bytes from 10.85.4.107: icmp_seq=1 ttl=64 time=0.794 ms
^C
--- 10.85.4.107 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.794/0.808/0.822/0.014 ms</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_internal_and_external_bridges_in_multiple_tenant_environment"><a class="anchor" href="#_internal_and_external_bridges_in_multiple_tenant_environment"></a>5.4. internal and external bridges in multiple tenant environment</h3>
<div class="paragraph">
<p>some highlights of the multiple VMX installation process we demonstrated:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>external management interfaces of all VMs from both instance share same
bridge: <code>br-ext</code></p>
</li>
<li>
<p>each instance has its own internal bridge <code>br-int-vmx1</code> and <code>br-int-vmx2</code> for
internal management interfaces of VCP/VFP VM</p>
</li>
<li>
<p>the physical NIC port p2p1 and p3p1 were each "split" into two virtual port
(VF), and each VF were used by a different instance.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These can be illustrated in this diagram:</p>
</div>
<div class="listingblock">
<div class="content">
<pre> vcp-vmx1                   br-int-vmx1 br-int-vmx2                   vcp-vmx2
+--------+                         |       |                          +--------+
|        |                         |       |                          |        |
|     em1+-------------------------+       +--------------------------+em1     |
|        |             vcp_int_vmx1|       |vcp_int_vmx2              |        |
| fxp0   |                         |       |                          | fxp0   |
+---+----+                         |       |                          +---+----+
    |               vfp-vmx1       |       |      vfp-vmx2                |
    |              +--------+      |       |      +--------+              |
    |              |   vfp_int-vmx1|       |vfp-int-vmx2   |              |
    |              |    eth0+------+       +------+eth0    |              |
    |              |        |-vmx1 |       |      |        |              |
    |              |eth1    |                     |  eth1  |              |
    |              ++-------+       p2p1          ++--+---++              |
    |               |              +--------+             |               |
    |               |              |        |             |               |
    |               |      --------+--VF0-- |             |               |
    |               |     ge-0/0/1 | -VF1---+------+      |               |
    |               |              |        | ge-0/0/1    |               |
    |               |              +--------+             |               |
    |               |                                     |               |
    |               |              p3p1                   |               |
    |               |              +--------+             |               |
    |               |     ge-0/0/0 |        |             |               |
    |               |      --------+--VF0-- | ge-0/0/0    |               |
    |               |              | -VF1---+---------    |               |
    |               |              |        |             |               |
    |               |              +--------+             |               |
    |vcp_ext-vmx1   | vfp_ext-vmx1            vfp_ext-vmx2|  vcp_ext-vmx2 |
+---+---------------+---+---------------------------------+---------------+----+
|                                 br-ext  br-ext-nic                           |
+---------------------------------------+--------------------------------------+
                                        |
                                        |em1</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_manually_uninstall_vmxs"><a class="anchor" href="#_manually_uninstall_vmxs"></a>5.5. manually uninstall VMXs</h3>
<div class="paragraph">
<p>the steps to "uninstall" VMX are much simpler:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>"poweroff" and remove VMs:</p>
<div class="literalblock">
<div class="content">
<pre>sudo virsh destroy vcp-vmx1
sudo virsh destroy vfp-vmx1
sudo virsh undefine vcp-vmx1
sudo virsh undefine vfp-vmx1</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo virsh destroy vcp-vmx2
sudo virsh destroy vfp-vmx2
sudo virsh undefine vcp-vmx2
sudo virsh undefine vfp-vmx2</pre>
</div>
</div>
</li>
<li>
<p>disable and delete the VNs and associated bridges:</p>
<div class="literalblock">
<div class="content">
<pre>sudo virsh net-destroy br-ext; \
sudo ifconfig em1 10.85.4.17/25; \
sudo route add default gw 10.85.4.1
sudo virsh net-undefine br-ext</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo virsh net-destroy br-int-vmx1
sudo virsh net-undefine br-int-vmx1
sudo virsh net-destroy br-int-vmx2
sudo virsh net-undefine br-int-vmx2</pre>
</div>
</div>
</li>
<li>
<p>disable VF of SR-IOV:</p>
<div class="literalblock">
<div class="content">
<pre>sudo rmmod ixgbevf; \
sudo rmmod ixgbe; \
sleep 5
sudo modprobe ixgbe</pre>
</div>
</div>
</li>
</ol>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_upgrading_vmx"><a class="anchor" href="#_upgrading_vmx"></a>6. upgrading VMX</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The procedure of upgrading VMX is conceptually similiar to the procedure of
upgrading physical Junos router - you load the new images and reboot the box
with them.</p>
</div>
<div class="paragraph">
<p>In practice, the steps are different. Since a standalone VMX uses <code>libvirt</code> to
manage the VMs, the upgrading steps involve modification of the corresponding
XML files that the libvirt tools rely on. These XML files provide all
parameters which provide a detail description about how a VM will be brought
up. the parameters include, but not limited to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>domain <code>name</code> - VM name</p>
</li>
<li>
<p>guest VM <code>memory</code></p>
</li>
<li>
<p>guest VM <code>vcpu</code> numbers</p>
</li>
<li>
<p>guest VM emulated cpu architecture</p>
</li>
<li>
<p>guest VM (emulated) hard disk images</p>
</li>
<li>
<p>guest VM bridges</p>
</li>
<li>
<p>guest VM (emulated) peripherals devices</p>
<div class="ulist">
<ul>
<li>
<p>video/audio/usb/serial/keyboard/mouse/etc&#8230;&#8203;</p>
</li>
</ul>
</div>
</li>
<li>
<p>guest VM (emulated) NICs</p>
</li>
<li>
<p>other devices/components of guest VM</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>with regard to guest OS upgrading, only "hard disk images" are related. So
essentially to upgrade VMX, only guest OS hard disk images (location) need to
be modified, followed by a guest VM reboot.</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>(optional) backup VCP/VFP VM configuration XML files:</p>
<div class="olist loweralpha">
<ol class="loweralpha" type="a">
<li>
<p>virsh dumpxml vfp-vmx1 &gt; vfp-vmx1.xml</p>
</li>
<li>
<p>virsh dumpxml vcp-vmx1 &gt; vcp-vmx1.xml</p>
</li>
<li>
<p>virsh net-dumpxml br-int-vmx1 &gt; br-int-vmx1.xml</p>
</li>
</ol>
</div>
</li>
<li>
<p>Edit the generated vfp-vmx1.xml &amp; vcp-vmx1.xml so they point to desired VCP
image, PFE image, and HDD image file.</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="title">VCP</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers"> 1</span><span class="tag">&lt;disk</span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">disk</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">file</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 2</span>  <span class="tag">&lt;driver</span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">directsync</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">qemu</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">qcow2</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers"> 3</span>  <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">/path/to/jinstall64-vmx-15.1F-20151104.0-domestic.img</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>   #<span class="tag">&lt;------</span>
<span class="line-numbers"> 4</span>  <span class="error">&lt;</span><span class="attribute-name">target</span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">ide</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">hda</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers"> 5</span>  <span class="tag">&lt;address</span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">controller</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">target</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">drive</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers"> 6</span><span class="tag">&lt;/disk&gt;</span>
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span><span class="tag">&lt;disk</span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">disk</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">file</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 9</span>  <span class="tag">&lt;driver</span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">directsync</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">qemu</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">qcow2</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers">10</span>  <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">/path/to/vmxhdd.img</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>     #<span class="tag">&lt;------</span>
<span class="line-numbers">11</span>  <span class="error">&lt;</span><span class="attribute-name">target</span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">ide</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">hdb</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers">12</span>  <span class="tag">&lt;address</span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">controller</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">target</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">drive</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">1</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers">13</span><span class="tag">&lt;/disk&gt;</span></code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">VFP</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers">1</span><span class="tag">&lt;disk</span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">disk</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">file</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>
<span class="line-numbers">2</span>  <span class="tag">&lt;driver</span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">directsync</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">qemu</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">raw</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers">3</span>  <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">/images/vmx_20151102.0/build/vmx1/images/vFPC-20151102.img</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>      #<span class="tag">&lt;------</span>
<span class="line-numbers">4</span>  <span class="error">&lt;</span><span class="attribute-name">target</span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">ide</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">hda</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>
<span class="line-numbers">5</span><span class="tag">&lt;/disk&gt;</span></code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>shutdown VCP/VFP VMs</p>
<div class="olist loweralpha">
<ol class="loweralpha" type="a">
<li>
<p>virsh destroy vfp-vmx1</p>
</li>
<li>
<p>virsh destroy vcp-vmx1</p>
</li>
<li>
<p>virsh undefine vfp-vmx1</p>
</li>
<li>
<p>virsh undefine vcp-vmx1</p>
</li>
<li>
<p>virsh net-undefine br-int-vmx1</p>
</li>
</ol>
</div>
</li>
<li>
<p>start VM with new configuration</p>
<div class="olist loweralpha">
<ol class="loweralpha" type="a">
<li>
<p>virsh define vfp-vmx1.xml</p>
</li>
<li>
<p>virsh define vcp-vmx1.xml</p>
</li>
<li>
<p>virsh net-define br-int-vmx1.xml</p>
</li>
<li>
<p>virsh net-start br-int-vmx1</p>
</li>
<li>
<p>virsh start vfp-vmx1</p>
</li>
<li>
<p>virsh start vcp-vmx1</p>
</li>
</ol>
</div>
</li>
</ol>
</div>
</div>
</div>
<div class="sect1">
<h2 id="troubleshooting-install"><a class="anchor" href="#troubleshooting-install"></a>7. troubleshooting installation script</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_ixgbe_compilation_error"><a class="anchor" href="#_ixgbe_compilation_error"></a>7.1. ixgbe compilation error</h3>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/images/vmx_20151102.0$ sudo ./vmx.sh -lvf --install
<span class="line-numbers"> 2</span>......
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>[OK]
<span class="line-numbers"> 5</span>Check IXGBE drivers...............................
<span class="line-numbers"> 6</span>[Command] cd /virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>[Command] rm -f ixgbe.ko
<span class="line-numbers"> 9</span>
<span class="line-numbers">10</span>[Command] make install
<span class="line-numbers">11</span>make -C /lib/modules/3.19.0-25-generic/build SUBDIRS=/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src modules
<span class="line-numbers">12</span>make[1]: Entering directory `/usr/src/linux-headers-3.19.0-25-generic'
<span class="line-numbers">13</span>  CC [M]  /virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.o
<span class="line-numbers">14</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c: In function ‘ixgbe_service_event_complete’:
<span class="line-numbers">15</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:339:2: error: implicit declaration of function ‘smp_mb__before_clear_bit’ [-Werror=implicit-function-declaration]
<span class="line-numbers">16</span>  smp_mb__before_clear_bit();
<span class="line-numbers">17</span>  ^
<span class="line-numbers">18</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c: In function ‘ixgbe_rx_hash’:
<span class="line-numbers">19</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:923:6: error: ‘struct sk_buff’ has no member named ‘rxhash’
<span class="line-numbers">20</span>   skb-&gt;rxhash = le32_to_cpu(rx_desc-&gt;wb.lower.hi_dword.rss);
<span class="line-numbers">21</span>      ^
<span class="line-numbers">22</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c: In function ‘ixgbe_del_mac_filter’:
<span class="line-numbers">23</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:4702:3: error: implicit declaration of function ‘compare_ether_addr’ [-Werror=implicit-function-declaration]
<span class="line-numbers">24</span>   if (!compare_ether_addr(addr, adapter-&gt;mac_table[i].addr) &amp;&amp;
<span class="line-numbers">25</span>   ^
<span class="line-numbers">26</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c: In function ‘ixgbe_ndo_bridge_getlink’:
<span class="line-numbers">27</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8800:2: error: too few arguments to function ‘ndo_dflt_bridge_getlink’
<span class="line-numbers">28</span>  return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode);
<span class="line-numbers">29</span>  ^
<span class="line-numbers">30</span>In file included from include/net/dst.h:13:0,
<span class="line-numbers">31</span>                 from include/net/sock.h:68,
<span class="line-numbers">32</span>                 from include/linux/tcp.h:22,
<span class="line-numbers">33</span>                 from /virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:40:
<span class="line-numbers">34</span>include/linux/rtnetlink.h:110:12: note: declared here
<span class="line-numbers">35</span> extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
<span class="line-numbers">36</span>            ^
<span class="line-numbers">37</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c: At top level:
<span class="line-numbers">38</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8830:2: error: unknown field ‘ndo_set_vf_tx_rate’ specified in initializer
<span class="line-numbers">39</span>  .ndo_set_vf_tx_rate = ixgbe_ndo_set_vf_bw,
<span class="line-numbers">40</span>  ^
<span class="line-numbers">41</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8830:2: warning: initialization from incompatible pointer type [enabled by default]
<span class="line-numbers">42</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8830:2: warning: (near initialization for ‘ixgbe_netdev_ops.ndo_set_vf_rate’) [enabled by default]
<span class="line-numbers">43</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8872:2: warning: initialization from incompatible pointer type [enabled by default]
<span class="line-numbers">44</span>  .ndo_fdb_add  = ixgbe_ndo_fdb_add,
<span class="line-numbers">45</span>  ^
<span class="line-numbers">46</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8872:2: warning: (near initialization for ‘ixgbe_netdev_ops.ndo_fdb_add’) [enabled by default]
<span class="line-numbers">47</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c: In function ‘ixgbe_ndo_bridge_getlink’:
<span class="line-numbers">48</span>/virtualization/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.c:8801:1: warning: control reaches end of non-void function [-Wreturn-type]
<span class="line-numbers">49</span> }
<span class="line-numbers">50</span> ^
<span class="line-numbers">51</span>cc1: some warnings being treated as errors
<span class="line-numbers">52</span>make[2]: *** [/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src/ixgbe_main.o] Error 1
<span class="line-numbers">53</span>make[1]: *** [_module_/images/vmx_20151102.0/drivers/ixgbe-3.19.1/src] Error 2
<span class="line-numbers">54</span>make[1]: Leaving directory `/usr/src/linux-headers-3.19.0-25-generic'
<span class="line-numbers">55</span>make: *** [default] Error 2
<span class="line-numbers">56</span>[Failed]
<span class="line-numbers">57</span>Log file........................................../images/vmx_20151102.0/build/vmx1/logs/vmx_1447967843.log
<span class="line-numbers">58</span>==================================================
<span class="line-numbers">59</span> Aborted!. 1 error(s) and 0 warning(s)
<span class="line-numbers">60</span>==================================================
<span class="line-numbers">61</span>ping@trinity:/images/vmx_20151102.0$
<span class="line-numbers">62</span>ping@trinity:/images/vmx_20151102.0$
<span class="line-numbers">63</span>ping@trinity:/images/vmx_20151102.0$</code></pre>
</div>
</div>
<div class="sect3">
<h4 id="_analysis_and_solution"><a class="anchor" href="#_analysis_and_solution"></a>7.1.1. analysis and solution</h4>
<div class="paragraph">
<p>This is ixgbe kernel driver module compilation error, and the cause is a
"wrong" kernel version.</p>
</div>
<div class="paragraph">
<p>Currently the solution is to change the host ubuntu kernel to 3.13, see section
<a href="#install-linux-kernel">install required linux kernel</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_82599_nic_not_recognized"><a class="anchor" href="#_82599_nic_not_recognized"></a>7.2. 82599 NIC not recognized</h3>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>......
<span class="line-numbers"> 2</span>[OK]
<span class="line-numbers"> 3</span>Setup huge pages to 32768.........................
<span class="line-numbers"> 4</span>[Command] echo 32768 &gt; /proc/sys/vm/nr_overcommit_hugepages
<span class="line-numbers"> 5</span>
<span class="line-numbers"> 6</span>[Command] mkdir /HugePage_vPFE
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>[Command] mount | grep &quot;HugePage_vPFE&quot;
<span class="line-numbers"> 9</span>
<span class="line-numbers">10</span>[Command] mount -t hugetlbfs hugetlbfs /HugePage_vPFE
<span class="line-numbers">11</span>[OK]
<span class="line-numbers">12</span>
<span class="line-numbers">13</span>[Command] cat /etc/apparmor.d/abstractions/libvirt-qemu  | grep HugePage_vPFE
<span class="line-numbers">14</span>
<span class="line-numbers">15</span>[Command] echo &quot;owner \&quot;/HugePage_vPFE/libvirt/qemu/**\&quot; rw,&quot; &gt;&gt; /etc/apparmor.d/abstractions/libvirt-qemu
<span class="line-numbers">16</span>
<span class="line-numbers">17</span>[Command] modprobe pci-stub
<span class="line-numbers">18</span>
<span class="line-numbers">19</span>[Command] echo &quot;8086 10ed&quot; &gt; /sys/bus/pci/drivers/pci-stub/new_id
<span class="line-numbers">20</span>
<span class="line-numbers">21</span>[Command] &gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">22</span>Number of Intel 82599 NICs........................0
<span class="line-numbers">23</span>Minimum Intel 82599 NICs available................No
<span class="line-numbers">24</span>==================================================
<span class="line-numbers">25</span>    Aborted!
<span class="line-numbers">26</span>==================================================</code></pre>
</div>
</div>
<div class="sect3">
<h4 id="_analysis_and_solution_2"><a class="anchor" href="#_analysis_and_solution_2"></a>7.2.1. analysis and solution</h4>
<div class="paragraph">
<p>the script does not find any of the specific NICs variants it was looking for:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>82599 ES</code></p>
</li>
<li>
<p><code>82599 EB</code></p>
</li>
<li>
<p><code>82599 EN</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>and so it aborted.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>66 vmx_system_setup_ixgbe()
67 {
68     nic_count=$(lspci | grep 82599 | grep 'ES\|EB\|EN' | wc -l)      #&lt;------
69     vmx_echo_textval "Number of Intel 82599 NICs" "$nic_count"
70     if [ $nic_count -eq 0 ];
71     then
72         vmx_echo_textval_red "Minimum Intel 82599 NICs available" "No"
73         vmx_echo_summary_banner "Aborted!"
74         exit 2
75     fi
76     vmx_echo_text "Configuring Intel 82599 Adapters for SRIOV"
77     cmd="rmmod ixgbevf"
78     vmx_exec_cmd -cmd "$cmd"
79     cmd="rmmod ixgbe"
80     vmx_exec_cmd -cmd "$cmd"
81     max_vfs_str="1"
82     nic_count=$(expr $nic_count - 1)
83     while [ $nic_count -gt 0 ];
84     do
85         max_vfs_str="$max_vfs_str,1"
86         nic_count=$(expr $nic_count - 1)
87     done
88     cmd="modprobe ixgbe max_vfs=$max_vfs_str"
89     vmx_exec_cmd -cmd "$cmd"
90     cmd="modprobe ixgbevf"
91     vmx_exec_cmd -cmd "$cmd"
92     cmd="modprobe tun"
93     vmx_exec_cmd -cmd "$cmd"
94     vmx_echo "[OK]"
95     cmd_descr="Number of Virtual Functions created"
96     cmd="lspci |grep 82599 | grep \"Virtual Function\" | wc -l"
97     vmx_exec_cmd -cmd "$cmd" -cmd_descr "$cmd_descr"
98 }</pre>
</div>
</div>
<div class="paragraph">
<p>The solution is to modify the installation script to ignore the <code>NS</code> <code>BS</code> <code>EN</code>
keyword when looking for <code>82599</code> NIC</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_hyperthread"><a class="anchor" href="#_hyperthread"></a>7.3. hyperthread</h3>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>==================================================
<span class="line-numbers"> 2</span>    System Setup Completed
<span class="line-numbers"> 3</span>==================================================
<span class="line-numbers"> 4</span>
<span class="line-numbers"> 5</span>[Command] brctl show br-ext | grep em1
<span class="line-numbers"> 6</span>can't get info No such device
<span class="line-numbers"> 7</span>Get Management Address of em1.....................
<span class="line-numbers"> 8</span>[Command] expr &quot;10.85.4.17&quot; != &quot;&quot;
<span class="line-numbers"> 9</span>1
<span class="line-numbers">10</span>[OK]
<span class="line-numbers">11</span>Generate libvirt files............................
<span class="line-numbers">12</span>[Command] python /virtualization/images/vmx_20151102.0/scripts/common/vmx_configure.py /virtualization/images/vmx_20151102.0/config/vmx.conf
<span class="line-numbers">13</span>Handling Host config
<span class="line-numbers">14</span>Initializing interface names for vmx1
<span class="line-numbers">15</span>Model=FPC
<span class="line-numbers">16</span>10.85.4.17
<span class="line-numbers">17</span>255.255.255.128
<span class="line-numbers">18</span>handling bridge config
<span class="line-numbers">19</span>Handling Routing Engine params
<span class="line-numbers">20</span>Handling Forwarding Engine params
<span class="line-numbers">21</span>['0', '1', '2', '3', '4', '5', '6', '7']
<span class="line-numbers">22</span>Core list HTO:
<span class="line-numbers">23</span>['0', '1', '2', '3', '4', '5', '6', '7']
<span class="line-numbers">24</span>Core list HT1:
<span class="line-numbers">25</span>[]
<span class="line-numbers">26</span>0
<span class="line-numbers">27</span>[Failed]
<span class="line-numbers">28</span>Traceback (most recent call last):
<span class="line-numbers">29</span>  File &quot;/images/vmx_20151102.0/scripts/common/vmx_configure.py&quot;, line 673, in &lt;module&gt;
<span class="line-numbers">30</span>    node_list[index].cpu_list[2*corenum + 1] = core_list_ht1[corenum]
<span class="line-numbers">31</span>IndexError: list index out of range
<span class="line-numbers">32</span>Log file........................................../images/vmx_20151102.0/build/vmx1/logs/vmx_1448050510.log
<span class="line-numbers">33</span>==================================================
<span class="line-numbers">34</span> Aborted!. 1 error(s) and 0 warning(s)
<span class="line-numbers">35</span>==================================================</code></pre>
</div>
</div>
<div class="sect3">
<h4 id="_analysis_and_solution_3"><a class="anchor" href="#_analysis_and_solution_3"></a>7.3.1. analysis and solution:</h4>
<div class="paragraph">
<p>starting for 15.x VMX start to support flow-cache which by design requires
hyperthreading feature.</p>
</div>
<div class="paragraph">
<p>this can be fixed by either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>enable hyperthread</p>
</li>
<li>
<p>change script to ignore hyperthreading checking</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="python"><span class="line-numbers"> 1</span><span class="integer">2134</span>     <span class="comment">#core_list_ht1 = core_list.cpu_list[num_physical_cores_per_node : num_physical_cores_per_node*2]</span>
<span class="line-numbers"> 2</span><span class="integer">2135</span>
<span class="line-numbers"> 3</span><span class="integer">2136</span>     <span class="keyword">print</span> <span class="string"><span class="delimiter">&quot;</span><span class="content">Core list HTO: </span><span class="delimiter">&quot;</span></span>
<span class="line-numbers"> 4</span><span class="integer">2137</span>     <span class="keyword">print</span> core_list_ht0
<span class="line-numbers"> 5</span><span class="integer">2138</span>     <span class="comment">#print &quot;Core list HT1: &quot;       </span><i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 6</span><span class="integer">2139</span>     <span class="comment">#print core_list_ht1           </span><i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 7</span><span class="integer">2140</span>
<span class="line-numbers"> 8</span><span class="integer">2141</span>     <span class="keyword">for</span> corenum <span class="keyword">in</span> <span class="predefined">range</span>(num_physical_cores_per_node):
<span class="line-numbers"> 9</span><span class="integer">2142</span>
<span class="line-numbers">10</span><span class="integer">2143</span>         <span class="keyword">print</span> corenum
<span class="line-numbers">11</span><span class="integer">2144</span>         <span class="comment">#node_list[index].cpu_list[2*corenum] = core_list_ht0[corenum]    </span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">12</span><span class="integer">2145</span>         node_list[index].cpu_list[corenum] = core_list_ht0[corenum]       <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">13</span><span class="integer">2146</span>         <span class="comment">#node_list[index].cpu_list[2*corenum + 1] = core_list_ht1[corenum]</span><i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">14</span><span class="integer">2147</span>
<span class="line-numbers">15</span><span class="integer">2148</span>         <span class="comment">#for corenum in range(len(node_list)):</span>
<span class="line-numbers">16</span><span class="integer">2149</span>         <span class="comment">#       node_list[index].cpu_list[corenum] =</span>
<span class="line-numbers">17</span><span class="integer">2150</span>     <span class="keyword">print</span> node_list[index].cpu_list</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>comment these lines</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>disable hyperthreading check</td>
</tr>
</table>
</div>
<div style="page-break-after: always;"></div>
</div>
</div>
</div>
</div>
<h1 id="_part_2_vmx_verification" class="sect0"><a class="anchor" href="#_part_2_vmx_verification"></a>Part 2: VMX verification</h1>
<div class="sect1">
<h2 id="_vcp_vfp_guest_vm_overview"><a class="anchor" href="#_vcp_vfp_guest_vm_overview"></a>8. VCP/VFP guest VM overview</h2>
<div class="sectionbody">
<div class="paragraph">
<p>After installation of VMX, at least two VMs should have been created:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>VM for virtual forwarding plane of VMX (VFP/vFPC)</p>
</li>
<li>
<p>VM for virtual control plane of VMX (VCP/vRE)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The VFP VM runs the (<code>yocto</code> based) virtual Trio forwarding plane software and
the VCP VM runs (<code>Freebsd</code> based) Junos OS.</p>
</div>
<div class="paragraph">
<p>This is demonstrated in the VMX architecture:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12179392/86ccb340-b544-11e5-9739-c926e0196ea9.png" alt="86ccb340 b544 11e5 9739 c926e0196ea9">
</div>
<div class="title">Figure 2. architecture of VMX instance</div>
</div>
<div class="paragraph">
<p>These VMs should be in "running" state, as can be listed with libvirt <code>virsh</code>
command below:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh list
[sudo] password for ping:
 Id    Name                           State
----------------------------------------------------
 2     vcp-vmx1                       running
 3     vfp-vmx1                       running</pre>
</div>
</div>
<div class="paragraph">
<p>The next thing we want to verify is the images these VM were built from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh domblklist 2
Target     Source
------------------------------------------------
hda        /virtualization/images/vmx_20151102.0/build/vmx1/images/jinstall64-vmx-15.1F-20151104.0-domestic.img
hdb        /virtualization/images/vmx_20151102.0/build/vmx1/images/vmxhdd.img
sda        /virtualization/images/vmx_20151102.0/images/metadata_usb.img</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh domblklist 3
Target     Source
------------------------------------------------
hda        /virtualization/images/vmx_20151102.0/build/vmx1/images/vFPC-20151102.img</pre>
</div>
</div>
<div class="paragraph">
<p>NOTE:
the libvirt <code>virsh</code> utility is a very powerful frontend toolset that can be
used to manage the VMs under its control <sup class="footnote">[<a id="_footnoteref_3" class="footnote" href="#_footnote_3" title="View footnote.">3</a>]</sup>.  There are a lot more other data can
be gathered simply from libvirt <code>virsh</code> tool, without the need to login to
guest VMs. see <a href="#VM-management">VM managent - virsh</a> section for more details
on this.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_login_to_the_vnc_console"><a class="anchor" href="#_login_to_the_vnc_console"></a>9. Login to the VNC console</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Right after a fresh VMX installation, the guest VCP/VFP VM will come up with no
configs. So the initial configuration (mgmt IP, GW, login, etc) needs to be
done via a "console" connection.  Previously we demonstrated how to login to
the vRE and vPFE <code>console</code> via <code>telnet</code>, which is also a convenient method to
collect the guest VM booting messages in the case that IP-based management
session is not available. Another way to acquire "console" of a guest VM is via
the built-in <code>VNC</code> support provided by KVM.</p>
</div>
<div class="paragraph">
<div class="title">locate the console/VNC access ports</div>
<p>A quick way to find out tcp port for console connection is to check <code>netstat</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity: $ sudo netstat -lntp
<span class="line-numbers"> 2</span>Active Internet connections (only servers)
<span class="line-numbers"> 3</span>Proto Recv-Q Send-Q Local Address  Foreign Address State   PID/Program name
<span class="line-numbers"> 4</span>tcp    0     0   127.0.0.1:5900    0.0.0.0:*       LISTEN  99035/qemu-system-x <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 5</span>tcp    0     0   127.0.0.1:5901    0.0.0.0:*       LISTEN  99192/qemu-system-x <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 6</span>tcp    0     0   127.0.0.1:8816    0.0.0.0:*       LISTEN  99035/qemu-system-x <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 7</span>tcp    0     0   127.0.0.1:8817    0.0.0.0:*       LISTEN  99192/qemu-system-x <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 8</span>tcp    0     0   10.85.4.17:53     0.0.0.0:*       LISTEN  98162/dnsmasq       <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers"> 9</span>tcp    0     0   192.168.122.1:53  0.0.0.0:*       LISTEN  1731/dnsmasq        <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">10</span>tcp    0     0   0.0.0.0:22        0.0.0.0:*       LISTEN  1438/sshd
<span class="line-numbers">11</span>tcp    0     0   127.0.0.1:6010    0.0.0.0:*       LISTEN  2677/1
<span class="line-numbers">12</span>tcp    0     0   127.0.0.1:6011    0.0.0.0:*       LISTEN  10570/2
<span class="line-numbers">13</span>tcp    0     0   127.0.0.1:6012    0.0.0.0:*       LISTEN  10636/3
<span class="line-numbers">14</span>tcp    0     0   127.0.0.1:6013    0.0.0.0:*       LISTEN  14223/4</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>VNC service port</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>console port</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>DNS service port <sup class="footnote">[<a id="_footnoteref_4" class="footnote" href="#_footnote_4" title="View footnote.">4</a>]</sup></td>
</tr>
</table>
</div>
<div class="paragraph">
<p>VNC ports can also be acquired via <code>virsh</code> command:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh vncdisplay 2
127.0.0.1:0

ping@trinity:~$ sudo virsh vncdisplay 3
127.0.0.1:1</pre>
</div>
</div>
<div class="paragraph">
<p>This indicates VNC port 5900 and 5901 for VCP and VFP respectively, same as
shown in previous <code>netstat</code> command output.</p>
</div>
<div class="paragraph">
<p>Now VNC GUI can be accessed via legacy <code>vncviewer</code> or <code>virt-viewer</code> tools, from
the server&#8217;s GUI.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12008990/fcfae4e4-ac32-11e5-9694-d735f90e9af7.png" alt="fcfae4e4 ac32 11e5 9694 d735f90e9af7">
</div>
<div class="title">Figure 3. example of using libvirt virt-viewer to access VNC console</div>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12009034/ce32a1b8-ac34-11e5-8b3d-117d84259194.png" alt="ce32a1b8 ac34 11e5 8b3d 117d84259194">
</div>
<div class="title">Figure 4. example of using linux vncviewer to access VNC console</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_ixgbe_driver_and_ixgbe_driven_interfaces"><a class="anchor" href="#_ixgbe_driver_and_ixgbe_driven_interfaces"></a>10. ixgbe driver and ixgbe-driven interfaces</h2>
<div class="sectionbody">
<div class="paragraph">
<p>With <code>SR-IOV</code> version of VMX installation, Juniper modified ixgbe driver will
be in use:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>ixgbe</code> kernel driver version is <code>3.19.1</code> (Juniper modified)</p>
</li>
<li>
<p>some more parameters are now supported than the default driver coming with
linux kernel</p>
</li>
<li>
<p><code>ixgbevf</code> driver kernel module is now loaded to support SR-IOV <code>Virtual
Function</code></p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ modinfo ixgbe
filename:       /lib/modules/3.13.0-32-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        3.19.1      #&lt;------
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, &lt;linux.nics@intel.com&gt;
srcversion:     B97B1E7CF79A25F5E4D7B96
alias:          pci:v00008086d00001560sv*sd*bc*sc*i*
alias:          pci:v00008086d00001558sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001557sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*
alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*
alias:          pci:v00008086d00001529sv*sd*bc*sc*i*
alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*
alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*
alias:          pci:v00008086d00001514sv*sd*bc*sc*i*
alias:          pci:v00008086d00001507sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*
alias:          pci:v00008086d00001517sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*
alias:          pci:v00008086d00001508sv*sd*bc*sc*i*
alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*
alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*
alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        dca
vermagic:       3.13.0-32-generic SMP mod_unload modversions
parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm:           MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm:           DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm:           RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm:           VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int)
parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm:           L2LBen:L2 Loopback Enable: 0 = disable, 1 = enable (default) (array of int)
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm:           LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm:           LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm:           LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm:           LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm:           LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm:           FdirPballoc:Flow Director packet buffer allocation level:
                        1 = 8k hash filters or 2k perfect filters
                        2 = 16k hash filters or 4k perfect filters
                        3 = 32k hash filters or 8k perfect filters (array of int)
parm:           AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm:           LRO:Large Receive Offload (0,1), default 1 = on (array of int)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)</pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>The ixgbe driver used here is the Juniper modified version in order to support
the ability of accepting ingress multicast packets arriving in a VF. So it&#8217;s
different than what is available from Intel website, even though both may show
same version number.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre>root@ubuntu1:~# modinfo ixgbevf
filename:       /lib/modules/3.13.0-32-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
version:        2.11.3-k
license:        GPL
description:    Intel(R) 82599 Virtual Function Driver
author:         Intel Corporation, &lt;linux.nics@intel.com&gt;
srcversion:     AE2D8A25951B508611E943D
alias:          pci:v00008086d00001515sv*sd*bc*sc*i*
alias:          pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
intree:         Y
vermagic:       3.13.0-32-generic SMP mod_unload modversions
signer:         Magrathea: Glacier signing key
sig_key:        5E:3C:0F:9C:A6:E3:65:43:53:5F:A2:BB:5B:70:9E:84:F1:6D:A7:C7
sig_hashalgo:   sha512
parm:           debug:Debug level (0=none,...,16=all) (int)</pre>
</div>
</div>
<div class="paragraph">
<p>To verify interfaces status from a linux server, the most commonly used
commands are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>ifconfig</code> command: "legacy" system administration utility in Unix-like
OS to manipulate or show interfaces</p>
</li>
<li>
<p><code>ip</code> command : relatively new commands, show / manipulate interface, routing,
devices, policy routing and tunnels, etc. much more features-rich than the
traditional <code>ifconfig</code> tool.</p>
</li>
<li>
<p><code>ethtool</code> command: to query or control network driver and low level hardware
settings, mostly used to polling L1 physical layer information of an
interface (e.g PCI address)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>in this section we&#8217;ll use <code>ifconfig</code> and <code>ip link show</code> to list the physical
ports, VFs and bridges.</p>
</div>
<div class="sect2">
<h3 id="_legacy_code_ifconfig_code_command"><a class="anchor" href="#_legacy_code_ifconfig_code_command"></a>10.1. legacy <code>ifconfig</code> command</h3>
<div class="paragraph">
<p>This is what it looks like after one VMX instance was built and brought up
to running status:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ ifconfig -a

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:37 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6468 (6.4 KB)  TX bytes:6468 (6.4 KB)

br-ext    Link encap:Ethernet  HWaddr 38:ea:a7:37:7c:54                 \     <i class="conum" data-value="1"></i><b>(1)</b>
          inet addr:10.85.4.17  Bcast:10.85.4.127  Mask:255.255.255.128  |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1             |
          RX packets:661 errors:0 dropped:0 overruns:0 frame:0           |
          TX packets:584 errors:0 dropped:0 overruns:0 carrier:0         |
          collisions:0 txqueuelen:0                                      |
          RX bytes:36305 (36.3 KB)  TX bytes:77528 (77.5 KB)             |
                                                                         |
br-ext-nic Link encap:Ethernet  HWaddr 52:54:00:9f:a0:77                 |    <i class="conum" data-value="2"></i><b>(2)</b>
          BROADCAST MULTICAST  MTU:1500  Metric:1                        |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0           |
          collisions:0 txqueuelen:500                                    \
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                          X br-ext bridge
                                                                         |  and I/Fs
vcp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:01               |    <i class="conum" data-value="3"></i><b>(3)</b>
          inet6 addr: fe80::fc04:17ff:fe01:101/64 Scope:Link             |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1             |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0           |
          collisions:0 txqueuelen:500                                    |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                         |
                                                                         |
vfp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:02               |    <i class="conum" data-value="4"></i><b>(4)</b>
          inet6 addr: fe80::fc04:17ff:fe01:102/64 Scope:Link             |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1             |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:85 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:500                                    |
          RX bytes:0 (0.0 B)  TX bytes:6934 (6.9 KB)                    /

br-int-vmx1 Link encap:Ethernet  HWaddr 52:54:00:ad:64:15               \     <i class="conum" data-value="5"></i><b>(5)</b>
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1             |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0           |
          collisions:0 txqueuelen:0                                      |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                         |
                                                                         |    <i class="conum" data-value="6"></i><b>(6)</b>
br-int-vmx1-nic Link encap:Ethernet  HWaddr 52:54:00:ad:64:15            |
          BROADCAST MULTICAST  MTU:1500  Metric:1                        |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0           |
          collisions:0 txqueuelen:500                                    \
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                          X
                                                                         |
vcp_int-vmx1 Link encap:Ethernet  HWaddr fe:54:00:84:52:fb               |    <i class="conum" data-value="7"></i><b>(7)</b>
          inet6 addr: fe80::fc54:ff:fe84:52fb/64 Scope:Link              |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1             |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0           |
          collisions:0 txqueuelen:500                                    |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                         |
                                                                         |
vfp_int-vmx1 Link encap:Ethernet  HWaddr fe:54:00:c0:ff:1f               |    <i class="conum" data-value="8"></i><b>(8)</b>
          inet6 addr: fe80::fc54:ff:fec0:ff1f/64 Scope:Link              |
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1             |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0             |
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0          |
          collisions:0 txqueuelen:500                                    |
          RX bytes:0 (0.0 B)  TX bytes:1376 (1.3 KB)                    /

virbr0    Link encap:Ethernet  HWaddr 2a:e1:60:a8:87:40                     \
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0 |
          UP BROADCAST MULTICAST  MTU:1500  Metric:1                         | <i class="conum" data-value="9"></i><b>(9)</b>
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                 |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0               |
          collisions:0 txqueuelen:0                                          |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                            /

em1       Link encap:Ethernet  HWaddr 38:ea:a7:37:7c:54          \
          inet6 addr: fe80::3aea:a7ff:fe37:7c54/64 Scope:Link     \
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1       |
          RX packets:1167 errors:0 dropped:1 overruns:0 frame:0    |
          TX packets:1071 errors:0 dropped:0 overruns:0 carrier:0  |
          collisions:0 txqueuelen:1000                             |
          RX bytes:80815 (80.8 KB)  TX bytes:129388 (129.3 KB)     |
                                                                   |
em2       Link encap:Ethernet  HWaddr 38:ea:a7:37:7c:55            |
          BROADCAST MULTICAST  MTU:1500  Metric:1                  |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                             |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                   |
                                                                   |
em9       Link encap:Ethernet  HWaddr 38:ea:a7:37:7b:d0            |
          BROADCAST MULTICAST  MTU:1500  Metric:1                  |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                             |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                   |
                                                                   |
em10      Link encap:Ethernet  HWaddr 38:ea:a7:37:7b:d1            |
          BROADCAST MULTICAST  MTU:1500  Metric:1                  |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                             |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                   |
                                                                   \
p2p1      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:a0             X <i class="conum" data-value="10"></i><b>(10)</b>
          inet6 addr: fe80::3aea:a7ff:fe17:65a0/64 Scope:Link      /
          UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST  MTU:2000| Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                             |
          RX bytes:810 (810.0 B)  TX bytes:558 (558.0 B)           |
                                                                   |
p2p2      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:a1            |
          BROADCAST MULTICAST  MTU:1500  Metric:1                  |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                             |
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                   |
                                                                   |
p3p1      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:84            |
          inet6 addr: fe80::3aea:a7ff:fe17:6584/64 Scope:Link      |
          UP BROADCAST RUNNING PROMISC ALLMULTI MULTICAST  MTU:2000| Metric:1
          RX packets:1 errors:0 dropped:1 overruns:0 frame:0       |
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                             |
          RX bytes:301 (301.0 B)  TX bytes:558 (558.0 B)           |
                                                                   |
p3p2      Link encap:Ethernet  HWaddr 38:ea:a7:17:65:85            |
          BROADCAST MULTICAST  MTU:1500  Metric:1                  |
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0       |
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0     |
          collisions:0 txqueuelen:1000                            /
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)                 /</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>the "external" bridge that bridges vRE (fxp0) and vPFE (ext) mgmt I/F to
the mgmt I/F of the "host" or server (em1)</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>br-ext-nic interface</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td><code>vcp_ext-vmx1</code>: external tap interface for <code>vcp-vmx1</code> fxp0 interface</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td><code>vfp_ext-vmx1</code>: external tap inferface for <code>vfp-vmx1</code> ext interface</td>
</tr>
<tr>
<td><i class="conum" data-value="5"></i><b>5</b></td>
<td>the "internal" bridge, that connects vRE (em1) and vPFE (eth1) for the
internal communications (software module download, IPC, PFE-RE host bound
traffic, etc)</td>
</tr>
<tr>
<td><i class="conum" data-value="6"></i><b>6</b></td>
<td><code>br-int-nic</code> interface</td>
</tr>
<tr>
<td><i class="conum" data-value="7"></i><b>7</b></td>
<td>internal tap interface for <code>vcp-vmx1</code> em1</td>
</tr>
<tr>
<td><i class="conum" data-value="8"></i><b>8</b></td>
<td>internal tap interface for <code>vfp-vmx1</code> eth1</td>
</tr>
<tr>
<td><i class="conum" data-value="9"></i><b>9</b></td>
<td>the default virtual network (VN), created when <code>libvirtd</code> daemon was first
installed and started. missing of this interface may indicate something wrong
with libvirt</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">the tap interface with "configured" MAC address</div>
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ ifconfig -a | grep 04:17
vcp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:01      <i class="conum" data-value="1"></i><b>(1)</b>
          inet6 addr: fe80::fc:04:17:ff:fe01:101/64 Scope:Link
vfp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:02      <i class="conum" data-value="2"></i><b>(2)</b>
          inet6 addr: fe80::fc:04:17:ff:fe01:102/64 Scope:Link</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>"peer" tap interface <sup class="footnoteref">[<a class="footnote" href="#_footnote_1" title="View footnote.">1</a>]</sup> for mgmt interface (fxp0) in VCP VM</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>"peer" tap interface for mgmt interface (ext) in VFP VM</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="__code_ip_code_tool_sr_iov"><a class="anchor" href="#__code_ip_code_tool_sr_iov"></a>10.2. <code>ip</code> tool (SR-IOV)</h3>
<div class="paragraph">
<p>comparing with legacy <code>ifconfig</code> command, the new <code>ip</code> tool is more powerful
- In this example it prints the configured SR-IOV VF info which <code>ifconfig</code> does
not provide, with much more interface properties.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/virtualization/images/vmx_20151102.0$ ip link show
1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10: virbr0: &lt;NO-CARRIER,BROADCAST,MULTICAST,UP&gt; mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 2a:e1:60:a8:87:40 brd ff:ff:ff:ff:ff:ff
27: em9: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:37:7b:d0 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
28: em10: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:37:7b:d1 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
29: p2p1: &lt;BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP&gt; mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:17:65:a0 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:04:17:01:02:02, tx rate 10000 (Mbps), spoof checking off, link-state auto               <i class="conum" data-value="1"></i><b>(1)</b>
30: p2p2: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:17:65:a1 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
31: em1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc mq master br-ext state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:37:7c:54 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
32: em2: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:37:7c:55 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
33: p3p1: &lt;BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP&gt; mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:17:65:84 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:04:17:01:02:01, tx rate 10000 (Mbps), spoof checking off, link-state aut                <i class="conum" data-value="2"></i><b>(2)</b>
34: p3p2: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:17:65:85 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
35: br-ext: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 38:ea:a7:37:7c:54 brd ff:ff:ff:ff:ff:ff
36: br-ext-nic: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc pfifo_fast master br-ext state DOWN mode DEFAULT group default qlen 500
    link/ether 52:54:00:9f:a0:77 brd ff:ff:ff:ff:ff:ff
37: br-int-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 52:54:00:ad:64:15 brd ff:ff:ff:ff:ff:ff
38: br-int-vmx1-nic: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc pfifo_fast master br-int-vmx1 state DOWN mode DEFAULT group default qlen 500
    link/ether 52:54:00:ad:64:15 brd ff:ff:ff:ff:ff:ff
39: vcp_ext-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master br-ext state UNKNOWN mode DEFAULT group default qlen 500
    link/ether fe:04:17:01:01:01 brd ff:ff:ff:ff:ff:ff
40: vcp_int-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master br-int-vmx1 state UNKNOWN mode DEFAULT group default qlen 500
    link/ether fe:54:00:84:52:fb brd ff:ff:ff:ff:ff:ff
41: vfp_ext-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master br-ext state UNKNOWN mode DEFAULT group default qlen 500
    link/ether fe:04:17:01:01:02 brd ff:ff:ff:ff:ff:ff
42: vfp_int-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master br-int-vmx1 state UNKNOWN mode DEFAULT group default qlen 500
    link/ether fe:54:00:c0:ff:1f brd ff:ff:ff:ff:ff:ff</pre>
</div>
</div>
<div class="listingblock">
<div class="title">the tap interface and VF with "configured" MAC address</div>
<div class="content">
<pre>29: p2p1: &lt;BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP&gt; mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:17:65:a0 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:04:17:01:02:02, tx rate 10000 (Mbps), spoof checking off, link-state auto               <i class="conum" data-value="1"></i><b>(1)</b>
33: p3p1: &lt;BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP&gt; mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 38:ea:a7:17:65:84 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 02:04:17:01:02:01, tx rate 10000 (Mbps), spoof checking off, link-state auto               <i class="conum" data-value="2"></i><b>(2)</b>
39: vcp_ext-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master br-ext state UNKNOWN mode DEFAULT group default qlen 500
    link/ether fe:04:17:01:01:01 brd ff:ff:ff:ff:ff:ff
41: vfp_ext-vmx1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast master br-ext state UNKNOWN mode DEFAULT group default qlen 500
    link/ether fe:04:17:01:01:02 brd ff:ff:ff:ff:ff:ff</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>p2p1 VF 0 L1/L2 info, this will map to the VMX router <code>ge-0/0/1</code> interface</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>p3p1 VF 0 L1/L2 info, this will map to the VMX router <code>ge-0/0/0</code> interface</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_host_interfaces_virtio"><a class="anchor" href="#_host_interfaces_virtio"></a>10.3. host interfaces (virtio)</h3>
<div class="paragraph">
<p>the list of interfaces after installation of an <code>virtio</code> version of VMX looks
very similiar to the ones after SR-IOV verions of VMX installation, but with
one exception - Besides external and internal bridges and the associated tap
interfaces for management, now we can see <strong>2 more tap interfaces generated by
virtio</strong>.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ge-0.0.0-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:02:01     \
<span class="line-numbers"> 2</span>          inet6 addr: fe80::fc04:17ff:fe01:201/64 Scope:Link     |
<span class="line-numbers"> 3</span>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     |
<span class="line-numbers"> 4</span>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0     |
<span class="line-numbers"> 5</span>          TX packets:401 errors:0 dropped:0 overruns:0 carrier:0 |
<span class="line-numbers"> 6</span>          collisions:0 txqueuelen:500                            \
<span class="line-numbers"> 7</span>          RX bytes:0 (0.0 B)  TX bytes:21084 (21.0 KB)            X <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 8</span>                                                                 |
<span class="line-numbers"> 9</span>ge-0.0.1-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:02:02      |
<span class="line-numbers">10</span>          inet6 addr: fe80::fc04:17ff:fe01:202/64 Scope:Link     |
<span class="line-numbers">11</span>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     |
<span class="line-numbers">12</span>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0     |
<span class="line-numbers">13</span>          TX packets:401 errors:0 dropped:0 overruns:0 carrier:0 |
<span class="line-numbers">14</span>          collisions:0 txqueuelen:500                            /
<span class="line-numbers">15</span>          RX bytes:0 (0.0 B)  TX bytes:21084 (21.0 KB)          /</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>virtio interfaces</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">"virtio interface" is tap interface</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo ethtool -i ge-0.0.0-vmx1
<span class="line-numbers"> 2</span>driver: tun
<span class="line-numbers"> 3</span>version: 1.6
<span class="line-numbers"> 4</span>firmware-version:
<span class="line-numbers"> 5</span>bus-info: tap   #&lt;------
<span class="line-numbers"> 6</span>supports-statistics: no
<span class="line-numbers"> 7</span>supports-test: no
<span class="line-numbers"> 8</span>supports-eeprom-access: no
<span class="line-numbers"> 9</span>supports-register-dump: no
<span class="line-numbers">10</span>supports-priv-flags: no</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">MAC address (virtio)</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/virtualization/images/vmx_20151102.0$ ifconfig -a | grep -i 04:17
<span class="line-numbers"> 2</span>ge-0.0.0-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:02:01
<span class="line-numbers"> 3</span>          inet6 addr: fe80::fc04:17ff:fe01:201/64 Scope:Link
<span class="line-numbers"> 4</span>ge-0.0.1-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:02:02
<span class="line-numbers"> 5</span>          inet6 addr: fe80::fc04:17ff:fe01:202/64 Scope:Link
<span class="line-numbers"> 6</span>vcp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:01
<span class="line-numbers"> 7</span>          inet6 addr: fe80::fc04:17ff:fe01:101/64 Scope:Link
<span class="line-numbers"> 8</span>vfp_ext-vmx1 Link encap:Ethernet  HWaddr fe:04:17:01:01:02
<span class="line-numbers"> 9</span>          inet6 addr: fe80::fc04:17ff:fe01:102/64 Scope:Link
<span class="line-numbers">10</span>virbr0    Link encap:Ethernet  HWaddr fe:04:17:01:02:01</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_vmx_fxp0_and_em1_interface"><a class="anchor" href="#_vmx_fxp0_and_em1_interface"></a>10.4. VMX fxp0 and em1 interface</h3>
<div class="paragraph">
<p>In a physical MX box, <code>fxp0</code> is used for the external Ethernet port on the RE,
while <code>em0</code>/<code>em1</code> (previously <code>fxp1</code>/<code>fxp2</code> in some old platforms) are internal
NICs on the RE that cross-connected to the internal 24x1GE switch on each SCB.</p>
</div>
<div class="paragraph">
<p>With a up and running VMX , we can now look at the <code>fxp0</code> and <code>em1</code> interface,
as if this is a physical MX router - with exactly the same Junos command.</p>
</div>
<div class="paragraph">
<p><code>fxp0</code> emulates the same physical <code>fxp0</code> NIC in MX RE board, and is still used
for the same purpose - serving managment connection from external device.</p>
</div>
<div class="listingblock">
<div class="title">VCP VM management interface - fxp0 (<code>external</code> interface)</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>root@vmx1&gt; show interfaces fxp0
<span class="line-numbers"> 2</span>Physical interface: fxp0, Enabled, Physical link is Up
<span class="line-numbers"> 3</span>  Interface index: 8, SNMP ifIndex: 1
<span class="line-numbers"> 4</span>  Type: Ethernet, Link-level type: Ethernet, MTU: 1514
<span class="line-numbers"> 5</span>  Device flags   : Present Running
<span class="line-numbers"> 6</span>  Interface flags: SNMP-Traps
<span class="line-numbers"> 7</span>  Current address: 02:04:17:01:01:01, Hardware address: 02:04:17:01:01:01
<span class="line-numbers"> 8</span>  Last flapped   : 2015-12-01 03:03:56 UTC (00:01:20 ago)
<span class="line-numbers"> 9</span>    Input packets : 384
<span class="line-numbers">10</span>    Output packets: 2
<span class="line-numbers">11</span>
<span class="line-numbers">12</span>  Logical interface fxp0.0 (Index 4) (SNMP ifIndex 13)
<span class="line-numbers">13</span>    Flags: Up SNMP-Traps 0x4000000 Encapsulation: ENET2
<span class="line-numbers">14</span>    Input packets : 169
<span class="line-numbers">15</span>    Output packets: 2
<span class="line-numbers">16</span>    Protocol inet, MTU: 1500
<span class="line-numbers">17</span>      Flags: Sendbcast-pkt-to-re, Is-Primary
<span class="line-numbers">18</span>      Addresses, Flags: Is-Default Is-Preferred Is-Primary
<span class="line-numbers">19</span>        Destination: 10.85.4.0/25, Local: 10.85.4.105, Broadcast: 10.85.4.127</code></pre>
</div>
</div>
<div class="paragraph">
<p>Although there is no real <code>CB</code> or <code>SCB</code> board in VMX, the interface <code>em1</code>
remains - it is now connecting directly to the only FPC - VCP VM. As in MX
router it can be viewed by the same <code>show interface</code> CLI:</p>
</div>
<div class="listingblock">
<div class="title">VCP VM interface - em1 (<code>internal</code> interface connecting to VFP only)</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>root# run show interfaces em1
<span class="line-numbers"> 2</span>Physical interface: em1, Enabled, Physical link is Up
<span class="line-numbers"> 3</span>  Interface index: 9, SNMP ifIndex: 23
<span class="line-numbers"> 4</span>  Type: Ethernet, Link-level type: Ethernet, MTU: 1514, Speed: 1000mbps
<span class="line-numbers"> 5</span>  Device flags   : Present Running
<span class="line-numbers"> 6</span>  Interface flags: SNMP-Traps
<span class="line-numbers"> 7</span>  Link type      : Full-Duplex
<span class="line-numbers"> 8</span>  Current address: 52:54:00:a8:60:36, Hardware address: 52:54:00:a8:60:36
<span class="line-numbers"> 9</span>  Last flapped   : Never
<span class="line-numbers">10</span>    Input packets : 41993
<span class="line-numbers">11</span>    Output packets: 41520
<span class="line-numbers">12</span>
<span class="line-numbers">13</span>  Logical interface em1.0 (Index 3) (SNMP ifIndex 24)
<span class="line-numbers">14</span>    Flags: Up SNMP-Traps 0x4000000 Encapsulation: ENET2
<span class="line-numbers">15</span>    Input packets : 41993
<span class="line-numbers">16</span>    Output packets: 41520
<span class="line-numbers">17</span>    Protocol inet, MTU: 1500
<span class="line-numbers">18</span>      Flags: Is-Primary
<span class="line-numbers">19</span>      Addresses, Flags: Is-Preferred
<span class="line-numbers">20</span>        Destination: 10/8, Local: 10.0.0.4, Broadcast: 10.255.255.255
<span class="line-numbers">21</span>      Addresses, Flags: Preferred Kernel Is-Preferred
<span class="line-numbers">22</span>        Destination: 128/2, Local: 128.0.0.1, Broadcast: 191.255.255.255
<span class="line-numbers">23</span>      Addresses, Flags: Primary Is-Default Is-Primary
<span class="line-numbers">24</span>        Destination: 128/2, Local: 128.0.0.4, Broadcast: 191.255.255.255
<span class="line-numbers">25</span>    Protocol inet6, MTU: 1500
<span class="line-numbers">26</span>    Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 0,
<span class="line-numbers">27</span>    Curr new hold cnt: 0, NH drop cnt: 0
<span class="line-numbers">28</span>      Flags: Is-Primary
<span class="line-numbers">29</span>      Addresses, Flags: Is-Preferred
<span class="line-numbers">30</span>        Destination: fe80::/64, Local: fe80::5254:ff:fea8:6036
<span class="line-numbers">31</span>      Addresses, Flags: Is-Default Is-Preferred Is-Primary
<span class="line-numbers">32</span>        Destination: fec0::/64, Local: fec0::a:0:0:4
<span class="line-numbers">33</span>    Protocol tnp, MTU: 1500
<span class="line-numbers">34</span>      Flags: Primary, Is-Primary
<span class="line-numbers">35</span>      Addresses
<span class="line-numbers">36</span>        Local: 0x4</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="__code_ge_x_y_z_code_interface"><a class="anchor" href="#__code_ge_x_y_z_code_interface"></a>10.5. <code>ge-x/y/z</code> interface</h3>
<div class="paragraph">
<p>the <code>ge-x/y/z</code> interface, built from SR-IOV VF or virtio tag interface in the
low level, represents the forwarding plane of VMX. The name <code>ge-</code> may not
be acurate - it can be a 10GE or even 100GE capable port, depending on the type
of (Intel) NIC card in use.</p>
</div>
<div class="listingblock">
<div class="title">ge-0/0/0 interface from Junos</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>labroot&gt; show interfaces ge-0/0/0
<span class="line-numbers"> 2</span>Physical interface: ge-0/0/0, Enabled, Physical link is Up
<span class="line-numbers"> 3</span>  Interface index: 139, SNMP ifIndex: 517
<span class="line-numbers"> 4</span>  Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode,
<span class="line-numbers"> 5</span>  Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None,
<span class="line-numbers"> 6</span>  Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled
<span class="line-numbers"> 7</span>  Pad to minimum frame size: Disabled
<span class="line-numbers"> 8</span>  Device flags   : Present Running
<span class="line-numbers"> 9</span>  Interface flags: SNMP-Traps Internal: 0x4000
<span class="line-numbers">10</span>  Link flags     : None
<span class="line-numbers">11</span>  CoS queues     : 8 supported, 8 maximum usable queues
<span class="line-numbers">12</span>  Current address: 02:04:17:01:02:01, Hardware address: 02:04:17:01:02:01
<span class="line-numbers">13</span>  Last flapped   : 2015-11-25 07:14:41 UTC (00:00:08 ago)
<span class="line-numbers">14</span>  Input rate     : 0 bps (0 pps)
<span class="line-numbers">15</span>  Output rate    : 0 bps (0 pps)
<span class="line-numbers">16</span>  Active alarms  : None
<span class="line-numbers">17</span>  Active defects : None
<span class="line-numbers">18</span>  Interface transmit statistics: Disabled</code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_virtual_networks_bridging_sr_iov"><a class="anchor" href="#_virtual_networks_bridging_sr_iov"></a>11. virtual networks/bridging (SR-IOV)</h2>
<div class="sectionbody">
<div class="paragraph">
<p>To list Virtual Networks (VN) generated by libvirt:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>    ping@trinity:~$ sudo virsh net-list
<span class="line-numbers">2</span>    [sudo] password for ping:
<span class="line-numbers">3</span>     Name                 State      Autostart     Persistent
<span class="line-numbers">4</span>    ----------------------------------------------------------
<span class="line-numbers">5</span>     br-ext               active     no            yes
<span class="line-numbers">6</span>     br-int-vmx1          active     no            yes
<span class="line-numbers">7</span>     default              active     yes           yes</code></pre>
</div>
</div>
<div class="paragraph">
<p>These virtual networks were constructed via linux bridges:</p>
</div>
<div class="listingblock">
<div class="title">VN <code>br-ext</code></div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers"> 1</span>sudo virsh net-dumpxml br-ext
<span class="line-numbers"> 2</span>
<span class="line-numbers"> 3</span><span class="tag">&lt;network&gt;</span>
<span class="line-numbers"> 4</span>  <span class="tag">&lt;name&gt;</span>br-ext<span class="tag">&lt;/name&gt;</span>
<span class="line-numbers"> 5</span>  <span class="tag">&lt;forward</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">route</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>                              <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 6</span>  <span class="tag">&lt;bridge</span> <span class="attribute-name">delay</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">br-ext</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">stp</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">on</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>           <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 7</span>  <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">52:54:00:9f:a0:77</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>                   <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers"> 8</span>  <span class="tag">&lt;ip</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">10.85.4.17</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">netmask</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">255.255.255.128</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>   <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers"> 9</span>    <span class="tag">&lt;dhcp&gt;</span>                                              <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">10</span>      <span class="tag">&lt;host</span> <span class="attribute-name">ip</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">10.85.4.105</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">mac</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:01</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">vcp-vmx1</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span> <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">11</span>      <span class="tag">&lt;host</span> <span class="attribute-name">ip</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">10.85.4.106</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">mac</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">02:04:17:01:01:02</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">vfp-vmx1</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span> <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">12</span>    <span class="tag">&lt;/dhcp&gt;</span>
<span class="line-numbers">13</span>  <span class="tag">&lt;/ip&gt;</span>
<span class="line-numbers">14</span><span class="tag">&lt;/network&gt;</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>here the VN <code>br-ext</code> is in "route" mode, traffic to/from this VN will be
"routed" to/from the host interface</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>the host interface is a bridge with same name <code>br-ext</code></td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>the host bridge&#8217;s MAC and IP info</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>IP address can be assigned based on MAC address from the libvirt built-in
DHCP server</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
currently VMX mgmt interface implementation doesn&#8217;t request IP address
acquirement via DHCP.  This may be changed in the future.
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">VN <code>br-int-vm1</code></div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers">1</span>sudo virsh net-dumpxml br-int-vmx1
<span class="line-numbers">2</span>
<span class="line-numbers">3</span><span class="tag">&lt;network&gt;</span>
<span class="line-numbers">4</span>  <span class="tag">&lt;name&gt;</span>br-int-vmx1<span class="tag">&lt;/name&gt;</span>
<span class="line-numbers">5</span>  <span class="tag">&lt;bridge</span> <span class="attribute-name">delay</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">br-int-vmx1</span><span class="delimiter">&quot;</span></span> <span class="attribute-name">stp</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">on</span><span class="delimiter">&quot;</span></span> <span class="tag">/&gt;</span>      #<span class="tag">&lt;------</span>
<span class="line-numbers">6</span><span class="error">&lt;</span><span class="error">/</span><span class="attribute-name">network</span><span class="tag">&gt;</span></code></pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p>the VN <code>br-int-vmx1</code> is an "isolated" network - there is no "forward"
attribute as presenting in <code>br-ext</code> VN</p>
</li>
<li>
<p>VN is implemented as a bridge with same name <code>br-int-vmx1</code></p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="title">the <code>default</code> VN</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh net-dumpxml default
<span class="line-numbers"> 2</span><span class="tag">&lt;network</span> <span class="attribute-name">connections</span>=<span class="string"><span class="delimiter">'</span><span class="content">2</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 3</span>  <span class="tag">&lt;name&gt;</span>default<span class="tag">&lt;/name&gt;</span>
<span class="line-numbers"> 4</span>  <span class="tag">&lt;uuid&gt;</span>35f8cee2-d217-4e1f-a4c0-e0e08c422a4e<span class="tag">&lt;/uuid&gt;</span>
<span class="line-numbers"> 5</span>  <span class="tag">&lt;forward</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">nat</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>                                  <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 6</span>    <span class="tag">&lt;nat&gt;</span>
<span class="line-numbers"> 7</span>      <span class="tag">&lt;port</span> <span class="attribute-name">start</span>=<span class="string"><span class="delimiter">'</span><span class="content">1024</span><span class="delimiter">'</span></span> <span class="attribute-name">end</span>=<span class="string"><span class="delimiter">'</span><span class="content">65535</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 8</span>    <span class="tag">&lt;/nat&gt;</span>
<span class="line-numbers"> 9</span>  <span class="tag">&lt;/forward&gt;</span>
<span class="line-numbers">10</span>  <span class="tag">&lt;bridge</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">virbr0</span><span class="delimiter">'</span></span> <span class="attribute-name">stp</span>=<span class="string"><span class="delimiter">'</span><span class="content">on</span><span class="delimiter">'</span></span> <span class="attribute-name">delay</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>            <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">11</span>  <span class="tag">&lt;ip</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">192.168.122.1</span><span class="delimiter">'</span></span> <span class="attribute-name">netmask</span>=<span class="string"><span class="delimiter">'</span><span class="content">255.255.255.0</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>  <i class="conum" data-value="3"></i><b>(3)</b>
<span class="line-numbers">12</span>    <span class="tag">&lt;dhcp&gt;</span>
<span class="line-numbers">13</span>      <span class="tag">&lt;range</span> <span class="attribute-name">start</span>=<span class="string"><span class="delimiter">'</span><span class="content">192.168.122.2</span><span class="delimiter">'</span></span> <span class="attribute-name">end</span>=<span class="string"><span class="delimiter">'</span><span class="content">192.168.122.254</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span> <i class="conum" data-value="4"></i><b>(4)</b>
<span class="line-numbers">14</span>    <span class="tag">&lt;/dhcp&gt;</span>
<span class="line-numbers">15</span>  <span class="tag">&lt;/ip&gt;</span>
<span class="line-numbers">16</span><span class="tag">&lt;/network&gt;</span></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>implemented as a bridge named <code>virbr0</code>,</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td><code>nat</code> is used for traffic forwarding</td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>IP address of <code>virbr0</code> bridge</td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>IP address pool for the libvirt built-in DHCP server.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Essentially, the default VN emulates a typical CPE router, via which the guest
VM can interact with external world without having to each has their own
interfaces configured with an external IP and routing entry, just like a home
router eliminates the need of having each home PC to acquire an external IP in
order to access the Internet.</p>
</div>
<div class="paragraph">
<p>the VN to brige binding relationship can also be verified via this virsh
command: <code>virsh net-info</code></p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh net-info default
<span class="line-numbers"> 2</span>Name:           default
<span class="line-numbers"> 3</span>UUID:           35f8cee2-d217-4e1f-a4c0-e0e08c422a4e
<span class="line-numbers"> 4</span>Active:         yes
<span class="line-numbers"> 5</span>Persistent:     yes
<span class="line-numbers"> 6</span>Autostart:      yes
<span class="line-numbers"> 7</span>Bridge:         virbr0
<span class="line-numbers"> 8</span>
<span class="line-numbers"> 9</span>ping@trinity:~$ sudo virsh net-info br-ext
<span class="line-numbers">10</span>Name:           br-ext
<span class="line-numbers">11</span>UUID:           b7abcd87-114e-40f3-8b4e-e913135aae56
<span class="line-numbers">12</span>Active:         yes
<span class="line-numbers">13</span>Persistent:     yes
<span class="line-numbers">14</span>Autostart:      no
<span class="line-numbers">15</span>Bridge:         br-ext
<span class="line-numbers">16</span>
<span class="line-numbers">17</span>ping@trinity:~$ sudo virsh net-info br-int-vmx1
<span class="line-numbers">18</span>Name:           br-int-vmx1
<span class="line-numbers">19</span>UUID:           36aa8f84-84e2-4bec-92cb-f2b41a4ed2f7
<span class="line-numbers">20</span>Active:         yes
<span class="line-numbers">21</span>Persistent:     yes
<span class="line-numbers">22</span>Autostart:      no
<span class="line-numbers">23</span>Bridge:         br-int-vmx1</code></pre>
</div>
</div>
<div class="paragraph">
<p>The bridge to interface binding relationship is as following:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>    ping@trinity:~$ brctl show
<span class="line-numbers"> 2</span>    bridge name        bridge id                STP enabled        interfaces
<span class="line-numbers"> 3</span>    br-ext                8000.38eaa7377c54        yes        br-ext-nic
<span class="line-numbers"> 4</span>                                                        em1
<span class="line-numbers"> 5</span>                                                        vcp_ext-vmx1
<span class="line-numbers"> 6</span>                                                        vfp_ext-vmx1
<span class="line-numbers"> 7</span>    br-int-vmx1                8000.525400b00308        yes        br-int-vmx1-nic
<span class="line-numbers"> 8</span>                                                        vcp_int-vmx1
<span class="line-numbers"> 9</span>                                                        vfp_int-vmx1
<span class="line-numbers">10</span>    virbr0                8000.000000000000        yes</code></pre>
</div>
</div>
<div class="paragraph">
<p>without login into the VM, VM virtual NIC info (SR-IOV) can be listed with
<code>virsh</code> command:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>    ping@trinity:~$ sudo virsh qemu-monitor-command vcp-vmx1 --hmp &quot;info network&quot;
<span class="line-numbers">2</span>    net0: index=0,type=nic,model=e1000,macaddr=02:04:17:01:01:01
<span class="line-numbers">3</span>     \ hostnet0: index=0,type=tap,fd=18
<span class="line-numbers">4</span>    net1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:8d:45:1e
<span class="line-numbers">5</span>     \ hostnet1: index=0,type=tap,fd=19</code></pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh qemu-monitor-command vfp-vmx1 --hmp "info network"
net0: index=0,type=nic,model=virtio-net-pci,macaddr=02:04:17:01:01:02
 \ hostnet0: index=0,type=tap,fd=18
net1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:5e:96:98
 \ hostnet1: index=0,type=tap,fd=21</pre>
</div>
</div>
<div class="sect2">
<h3 id="_mapping_of_bridge_tap_guest_vnic_sr_iov"><a class="anchor" href="#_mapping_of_bridge_tap_guest_vnic_sr_iov"></a>11.1. mapping of bridge, tap, guest vNIC (SR-IOV)</h3>
<div class="paragraph">
<p>this diagram illustrated the relationship between these interfaces:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>external/internal mgmt interfaces in VCP VM: fxp0, em1</p>
</li>
<li>
<p>external/internal mgmt interfaces in VFP VM: eth0(ext), eth1(int)</p>
</li>
<li>
<p>qemu tap interfaces: vcp(vfp)_ext(int)-vmx1</p>
</li>
<li>
<p>linux bridge interface: br-ext, br-int-vmx1</p>
<div class="literalblock">
<div class="content">
<pre> vcp-vmx1                                   br-int-vmx1
+--------+                                  |52:54:00:b0:03:08
|     em1|52:54:00:8d:45:1e                 |
|        +-------------------------+--------+
|        |                     vcp_int-vmx1 |fe:54:00:8d:45:1e
| fxp0   |                                  |
+---+----+                                  |
    |02:04:17:01:01:01                      |
    |              +--------+               |
    |              |vfp-vmx1|   vfp_int-vmx1|fe:54:00:5e:96:98
    |              |    eth1+---------------+
    |              |      |52:54:00:5e:96:98|
    |              |  eth0  |               |
    |              +----+---+               |
    |                   |02:04:17:01:01:02
    |                   |
    |                   |
    |                   |
    |                   |
    |                   |
    |fe:04:17:01:01:01  |02:04:17:01:01:02
    |vcp_ext-vmx1       |vfp_ext-vmx1
----+---------------+---+-------------+-----br-ext
                    |38:ea:a7:37:7c:54| (mac configured
                    |                 |   same as em1)
                    |em1              |br-ext-nic (a0:77)</pre>
</div>
</div>
</li>
<li>
<p>brige <code>br-int-vmx1</code> , with attached qemu tap interfaces vNIC <code>vcp_int-vmx1</code>
and <code>vfp_int-vmx1</code> , and the corresponding guest VM interfaces <code>em1</code> and <code>int</code>,
are for internal communication only, no packets will exit to outside
networks, so MAC address does not matters</p>
</li>
<li>
<p>brige <code>br-ext</code> clones MAC and IP from host mgmt port, so it represents the server
from the external networks' point of view.</p>
</li>
<li>
<p>VCP(RE) VM mgmt port <code>fxp0</code> and VFP(vFPC) VM mgmt port <code>ext</code> or <code>eth0</code> will
communicate with external network, so they will expose to the outside network
and therefore the MAC addresses need to be unique in the segment.</p>
</li>
<li>
<p>packets of <code>fxp0</code> goes to external network via this path:</p>
<div class="ulist">
<ul>
<li>
<p>packet exits <code>fxp0</code> of guest VM <code>vcp-vmx1</code></p>
</li>
<li>
<p>packet is received from host tap interface <code>vcp_ext-vmx1</code></p>
</li>
<li>
<p>packet is forwarded to physical mgmt interface <code>em1</code> via bridge <code>br-ext</code></p>
</li>
<li>
<p>packet is forwarded to external networks</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<table class="tableblock frame-all grid-all" style="width: 80%;">
<caption class="title">Table 4. bridge/VN/tap/guest interface mapping table</caption>
<colgroup>
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">bridge/VN</th>
<th class="tableblock halign-left valign-top">forward mode</th>
<th class="tableblock halign-left valign-top">qemu tap interface</th>
<th class="tableblock halign-left valign-top">guest VM interface</th>
<th class="tableblock halign-left valign-top">guest VM</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" rowspan="2"><p class="tableblock">br-int-vmx1</p></td>
<td class="tableblock halign-left valign-top" rowspan="2"><p class="tableblock">none</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>vcp_int-vmx1</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>em1</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VCP(VRE)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>vfp_int-vmx1</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>int</code> or <code>eth1</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VFP(VPFE)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="2"><p class="tableblock">br-ext-vmx1</p></td>
<td class="tableblock halign-left valign-top" rowspan="2"><p class="tableblock">routed</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>vcp_ext-vmx1</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>fxp0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VCP(VRE)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>vfp_ext-vmx1</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>ext</code> or <code>eth0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VFP(VPFE)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">virbr0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">nat</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="VCPU_ESSENTIAL"><a class="anchor" href="#VCPU_ESSENTIAL"></a>12. vcpu essential</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In <code>vmx.conf</code> file of previous example, 4 CPUs were allocated to vPFE VM for
"lite mode":</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>FORWARDING_PLANE:
<span class="line-numbers">2</span>    memory-mb   : 16384
<span class="line-numbers">3</span>    vcpus       : 4</code></pre>
</div>
</div>
<div class="paragraph">
<p>In another example we have 16 CPUs allocated for "performance mode":</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>FORWARDING_PLANE:
<span class="line-numbers">2</span>    memory-mb   : 16384
<span class="line-numbers">3</span>    vcpus       : 4</code></pre>
</div>
</div>
<div class="paragraph">
<p>After logging into vPFE VM we can check the guest VM CPU properties:</p>
</div>
<div class="listingblock">
<div class="title">vfp</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>root@localhost:~# lscpu
<span class="line-numbers"> 2</span>Architecture:          x86_64
<span class="line-numbers"> 3</span>CPU op-mode(s):        32-bit, 64-bit
<span class="line-numbers"> 4</span>Byte Order:            Little Endian
<span class="line-numbers"> 5</span>CPU(s):                16       #&lt;------
<span class="line-numbers"> 6</span>On-line CPU(s) list:   0-15
<span class="line-numbers"> 7</span>Thread(s) per core:    1
<span class="line-numbers"> 8</span>Core(s) per socket:    16       #&lt;------
<span class="line-numbers"> 9</span>Socket(s):             1
<span class="line-numbers">10</span>NUMA node(s):          1
<span class="line-numbers">11</span>Vendor ID:             GenuineIntel
<span class="line-numbers">12</span>CPU family:            6
<span class="line-numbers">13</span>Model:                 42
<span class="line-numbers">14</span>Model name:            Intel Xeon E312xx (Sandy Bridge)
<span class="line-numbers">15</span>Stepping:              1
<span class="line-numbers">16</span>CPU MHz:               2992.582
<span class="line-numbers">17</span>BogoMIPS:              5985.16
<span class="line-numbers">18</span>Virtualization:        VT-x     #&lt;------
<span class="line-numbers">19</span>Hypervisor vendor:     KVM      #&lt;------
<span class="line-numbers">20</span>Virtualization type:   full
<span class="line-numbers">21</span>L1d cache:             32K
<span class="line-numbers">22</span>L1i cache:             32K
<span class="line-numbers">23</span>L2 cache:              4096K
<span class="line-numbers">24</span>NUMA node0 CPU(s):     0-15
<span class="line-numbers">25</span>
<span class="line-numbers">26</span>root@localhost:~# cat /proc/cpuinfo
<span class="line-numbers">27</span>processor       : 0
<span class="line-numbers">28</span>vendor_id       : GenuineIntel
<span class="line-numbers">29</span>cpu family      : 6
<span class="line-numbers">30</span>model           : 42
<span class="line-numbers">31</span>model name      : Intel Xeon E312xx (Sandy Bridge)
<span class="line-numbers">32</span>stepping        : 1
<span class="line-numbers">33</span>microcode       : 0x1
<span class="line-numbers">34</span>cpu MHz         : 2992.582
<span class="line-numbers">35</span>cache size      : 4096 KB
<span class="line-numbers">36</span>physical id     : 0
<span class="line-numbers">37</span>siblings        : 16
<span class="line-numbers">38</span>core id         : 0
<span class="line-numbers">39</span>cpu cores       : 16
<span class="line-numbers">40</span>apicid          : 0
<span class="line-numbers">41</span>initial apicid  : 0
<span class="line-numbers">42</span>fpu             : yes
<span class="line-numbers">43</span>fpu_exception   : yes
<span class="line-numbers">44</span>cpuid level     : 13
<span class="line-numbers">45</span>wp              : yes
<span class="line-numbers">46</span>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
<span class="line-numbers">47</span>                  pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb
<span class="line-numbers">48</span>                  rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq
<span class="line-numbers">49</span>                  vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt
<span class="line-numbers">50</span>                  tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
<span class="line-numbers">51</span>                  lahf_lm xsa veopt vnmi ept fsgsbase smep erms
<span class="line-numbers">52</span>bogomips        : 5985.16
<span class="line-numbers">53</span>clflush size    : 64
<span class="line-numbers">54</span>cache_alignment : 64
<span class="line-numbers">55</span>address sizes   : 40 bits physical, 48 bits virtual
<span class="line-numbers">56</span>power management:
<span class="line-numbers">57</span>......</code></pre>
</div>
</div>
<div class="paragraph">
<p>The <code>Hypervisor vendor</code> indicates the type of hypervisor that the current vCPU
was emulated from. In this case it&#8217;s KVM emulated CPU. In the case of other
hypervisor this option will change accordingly. below is an sample taken from
a vmware environment:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>Architecture:          x86_64
<span class="line-numbers"> 2</span>CPU op-mode(s):        32-bit, 64-bit
<span class="line-numbers"> 3</span>Byte Order:            Little Endian
<span class="line-numbers"> 4</span>CPU(s):                8
<span class="line-numbers"> 5</span>On-line CPU(s) list:   0-7
<span class="line-numbers"> 6</span>Thread(s) per core:    1
<span class="line-numbers"> 7</span>Core(s) per socket:    8
<span class="line-numbers"> 8</span>Socket(s):             1
<span class="line-numbers"> 9</span>NUMA node(s):          1
<span class="line-numbers">10</span>Vendor ID:             GenuineIntel
<span class="line-numbers">11</span>CPU family:            6
<span class="line-numbers">12</span>Model:                 47
<span class="line-numbers">13</span>Stepping:              2
<span class="line-numbers">14</span>CPU MHz:               2394.000
<span class="line-numbers">15</span>BogoMIPS:              4788.00
<span class="line-numbers">16</span>Hypervisor vendor:     VMware
<span class="line-numbers">17</span>Virtualization type:   full
<span class="line-numbers">18</span>L1d cache:             32K
<span class="line-numbers">19</span>L1i cache:             32K
<span class="line-numbers">20</span>L2 cache:              256K
<span class="line-numbers">21</span>L3 cache:              30720K
<span class="line-numbers">22</span>NUMA node0 CPU(s):     0-7</code></pre>
</div>
</div>
<div class="paragraph">
<p>for comparison purpose, the original cpu info from the host server is also
listed here:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@ubuntu1:~$ lscpu
<span class="line-numbers"> 2</span>Architecture:          x86_64
<span class="line-numbers"> 3</span>CPU op-mode(s):        32-bit, 64-bit
<span class="line-numbers"> 4</span>Byte Order:            Little Endian
<span class="line-numbers"> 5</span>CPU(s):                20
<span class="line-numbers"> 6</span>On-line CPU(s) list:   0-19
<span class="line-numbers"> 7</span>Thread(s) per core:    1
<span class="line-numbers"> 8</span>Core(s) per socket:    10
<span class="line-numbers"> 9</span>Socket(s):             2
<span class="line-numbers">10</span>NUMA node(s):          2
<span class="line-numbers">11</span>Vendor ID:             GenuineIntel
<span class="line-numbers">12</span>CPU family:            6
<span class="line-numbers">13</span>Model:                 62
<span class="line-numbers">14</span>Stepping:              4
<span class="line-numbers">15</span>CPU MHz:               2992.939
<span class="line-numbers">16</span>BogoMIPS:              6000.66
<span class="line-numbers">17</span>Virtualization:        VT-x
<span class="line-numbers">18</span>L1d cache:             32K
<span class="line-numbers">19</span>L1i cache:             32K
<span class="line-numbers">20</span>L2 cache:              256K
<span class="line-numbers">21</span>L3 cache:              25600K
<span class="line-numbers">22</span>NUMA node0 CPU(s):     0-9
<span class="line-numbers">23</span>NUMA node1 CPU(s):     10-19
<span class="line-numbers">24</span>
<span class="line-numbers">25</span>ping@ubuntu1:~$ cat /proc/cpuinfo
<span class="line-numbers">26</span>processor       : 0
<span class="line-numbers">27</span>vendor_id       : GenuineIntel
<span class="line-numbers">28</span>cpu family      : 6
<span class="line-numbers">29</span>model           : 62
<span class="line-numbers">30</span>model name      : Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
<span class="line-numbers">31</span>stepping        : 4
<span class="line-numbers">32</span>microcode       : 0x427
<span class="line-numbers">33</span>cpu MHz         : 2992.939
<span class="line-numbers">34</span>cache size      : 25600 KB
<span class="line-numbers">35</span>physical id     : 0
<span class="line-numbers">36</span>siblings        : 10
<span class="line-numbers">37</span>core id         : 0
<span class="line-numbers">38</span>cpu cores       : 10
<span class="line-numbers">39</span>apicid          : 0
<span class="line-numbers">40</span>initial apicid  : 0
<span class="line-numbers">41</span>fpu             : yes
<span class="line-numbers">42</span>fpu_exception   : yes
<span class="line-numbers">43</span>cpuid level     : 13
<span class="line-numbers">44</span>wp              : yes
<span class="line-numbers">45</span>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
<span class="line-numbers">46</span>                  pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
<span class="line-numbers">47</span>                  syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs
<span class="line-numbers">48</span>                  bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu
<span class="line-numbers">49</span>                  pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3
<span class="line-numbers">50</span>                  cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
<span class="line-numbers">51</span>                  tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat
<span class="line-numbers">52</span>                  epb xsaveopt pln pts dtherm tpr_shadow vnmi
<span class="line-numbers">53</span>                  flexpriority ept vpid fsgsbase smep erms
<span class="line-numbers">54</span>bogomips        : 5985.87
<span class="line-numbers">55</span>clflush size    : 64
<span class="line-numbers">56</span>cache_alignment : 64
<span class="line-numbers">57</span>address sizes   : 46 bits physical, 48 bits virtual
<span class="line-numbers">58</span>power management:</code></pre>
</div>
</div>
<div class="paragraph">
<p>from the comparison it is noticed that the CPU features set (flags) are not
identical between the host and guest OS. the host CPU features were provided by
the CPU hardware capability and OS support, while the guest CPU features can be
either emulated by QEMU software, or simly "exposed" directly from the host CPU
capability, by KVM. This was started in libvirt in the form of XML format (see
<a href="#VIRSH_CAP"><code>virsh capabilities</code></a>), and passed to <code>qemu-system-x86_64</code> process
in the form of CPU parameter list (<code>-cpu</code>) before the guess VM was brought up.</p>
</div>
<div id="VMX-qemu-process" class="listingblock">
<div class="title">VMX qemu processes</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/virtualization/images/vmx_20151102.0/config$ ps -ef | grep -i qemu
<span class="line-numbers"> 2</span>root      17757      1 18 19:01 ?        00:11:45
<span class="line-numbers"> 3</span>/usr/bin/qemu-system-x86_64 -name vcp-vmx1 -S -machine
<span class="line-numbers"> 4</span>pc-0.13,accel=kvm,usb=off -cpu
<span class="line-numbers"> 5</span>SandyBridge,+invtsc,+erms,+smep,+fsgsbase,+pdpe1gb,+rdrand,+f16c,+osxsave,+
<span class="line-numbers"> 6</span>dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
<span class="line-numbers"> 7</span>-m 1954 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
<span class="line-numbers"> 8</span>79ab0bbf-b63a-4a08-87ce-ad7670186754 -smbio s type=0,vendor=Juniper -smbios
<span class="line-numbers"> 9</span>type=1,manufacturer=VMX,product=VM-vcp_vmx1-161-re-0,version=0.1.0
<span class="line-numbers">10</span>-no-user-config -nodefaults -chardev
<span class="line-numbers">11</span>socket,id=charmonitor,path=//lib/libvirt/qemu/vcp-vmx1.monitor,server,nowai
<span class="line-numbers">12</span>t -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
<span class="line-numbers">13</span>-no-shutdown -boot strict=on -device
<span class="line-numbers">14</span>piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
<span class="line-numbers">15</span>file=/virtualization/images/vmx_20151102.0/build/vmx1/ima
<span class="line-numbers">16</span>ges/jinstall64-vmx-15.1F-20151104.0-domestic.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=directsync
<span class="line-numbers">17</span>-device
<span class="line-numbers">18</span>ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive
<span class="line-numbers">19</span>file=/virtualization/i
<span class="line-numbers">20</span>mages/vmx_20151102.0/build/vmx1/images/vmxhdd.img,if=none,id=drive-ide0-0-1,format=qcow2,cache=directsync
<span class="line-numbers">21</span>-device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive
<span class="line-numbers">22</span>file=/virtualization/images/vmx_2
<span class="line-numbers">23</span>0151102.0/images/metadata_usb.img,if=none,id=drive-usb-disk0,format=raw,cache=directsync
<span class="line-numbers">24</span>-device usb-storage,drive=drive-usb-disk0,id=usb-disk0,removable=off
<span class="line-numbers">25</span>-netdev tap,fd=18,id=hostnet0 -device e1000,netdev=ho
<span class="line-numbers">26</span>stnet0,id=net0,mac=02:04:17:01:01:01,bus=pci.0,addr=0x3 -netdev
<span class="line-numbers">27</span>tap,fd=19,id=hostnet1,vhost=on,vhostfd=20 -device
<span class="line-numbers">28</span>virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:84:52:fb,bus=pci.0,addr=0x5
<span class="line-numbers">29</span>-chardev socket,id=charserial0,host=127.0.0.1,port=8816,telnet,server,nowait -device
<span class="line-numbers">30</span>isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc
<span class="line-numbers">31</span>127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
<span class="line-numbers">32</span>AC97,id=sound0,bus=pci.0,addr=0x4 -device
<span class="line-numbers">33</span>virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
<span class="line-numbers">34</span>
<span class="line-numbers">35</span>root      17880      1 62 19:01 ?        00:40:31
<span class="line-numbers">36</span>/usr/bin/qemu-system-x86_64 -name vfp-vmx1 -S -machine
<span class="line-numbers">37</span>pc-i440fx-trusty,accel=kvm,usb=off,mem-merge=off
<span class="line-numbers">38</span>-cpu SandyBridge,+invtsc,+erms,+smep,+fsgsbase,+pdpe1gb,+
<span class="line-numbers">39</span>rdrand,+f16c,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
<span class="line-numbers">40</span>-m 15625 -mem-prealloc -mem-path /HugePage_vPFE/libvirt/qemu -realtime
<span class="line-numbers">41</span>mlock=off -smp 4,s ockets=1,cores=4,threads=1 -uuid
<span class="line-numbers">42</span>5d7f31e7-b678-4605-973f-0cb6a3e3b8c1 -no-user-config -nodefaults -chardev
<span class="line-numbers">43</span>socket,id=charmonitor,path=//lib/libvirt/qemu/vfp-vmx1.monitor,server,nowait
<span class="line-numbers">44</span>-mon chardev=charmonitor,id =monitor,mode=control -rtc base=utc
<span class="line-numbers">45</span>-no-shutdown -boot strict=on -device
<span class="line-numbers">46</span>piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
<span class="line-numbers">47</span>file=/virtualization/images/vmx_20151102.0/build/vmx1/images/vFPC-20151102.img,if=none
<span class="line-numbers">48</span>,id=drive-ide0-0-0,format=raw,cache=directsync -device
<span class="line-numbers">49</span>ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
<span class="line-numbers">50</span>-netdev tap,fd=18,id=hostnet0,vhost=on,vhostfd=20 -device
<span class="line-numbers">51</span>virtio-net-pci,netdev=hostnet
<span class="line-numbers">52</span>0,id=net0,mac=02:04:17:01:01:02,bus=pci.0,addr=0x3 -netdev
<span class="line-numbers">53</span>tap,fd=21,id=hostnet1,vhost=on,vhostfd=22 -device
<span class="line-numbers">54</span>virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:c0:ff:1f,bus=pci.0,addr=0x4
<span class="line-numbers">55</span>-chardev socket,id=charserial0,host=127.0.0.1,port=8817,telnet,server,nowait -device
<span class="line-numbers">56</span>isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc
<span class="line-numbers">57</span>127.0.0.1:1 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device AC97
<span class="line-numbers">58</span>,id=sound0,bus=pci.0,addr=0x5 -device
<span class="line-numbers">59</span>pci-assign,configfd=23,host=23:10.0,id=hostdev0,bus=pci.0,addr=0x6 -device
<span class="line-numbers">60</span>pci-assign,configfd=24,host=06:10.0,id=hostdev1,bus=pci.0,addr=0x7 -device
<span class="line-numbers">61</span>virtio-balloon-pci,id=b alloon0,bus=pci.0,addr=0x8 -msg timestamp=on</code></pre>
</div>
</div>
<div class="paragraph">
<p>Here are some highlights about QEMU processes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>/usr/bin/qemu-system-x86_64</code> is the QEMU command to start a VM process</p>
</li>
<li>
<p>the long list of QEMU parameters describe how the process will run and what
resources will be allocated to it</p>
</li>
<li>
<p>the break-down of all these QEMU parameters will be covered later (TODO)</p>
</li>
<li>
<p>the VCP(RE) and VFP(FPC) VM are each running as a QEMU process.</p>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ ps -ef | grep qemu | cut -c -105
root      42025      1 18 Nov23 ?        02:40:58 /usr/bin/qemu-system-x86_64 -name vcp-vmx1 -S -machine
root      42148      1 63 Nov23 ?        09:24:41 /usr/bin/qemu-system-x86_64 -name vfp-vmx1 -S -machine</pre>
</div>
</div>
</li>
<li>
<p>each VCPU(Virtual CPU) is essentially just a "thread" running inside one of
the QEMU processes</p>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="title">threads (vCPU):</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh qemu-monitor-command 2 --hmp &quot;info cpus&quot;
<span class="line-numbers"> 2</span>* CPU #0: pc=0xffffffff80921b83 thread_id=42028             <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>ping@trinity:~$ sudo virsh qemu-monitor-command 3 --hmp &quot;info cpus&quot;
<span class="line-numbers"> 5</span>* CPU #0: pc=0xffffffff8100af50 (halted) thread_id=42151    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 6</span>  CPU #1: pc=0xffffffff8100af50 (halted) thread_id=42152    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 7</span>  CPU #2: pc=0xffffffff8100af50 (halted) thread_id=42153    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 8</span>  CPU #3: pc=0xffffffff8100af50 (halted) thread_id=42154    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers"> 9</span>
<span class="line-numbers">10</span>ping@trinity:~$ ps -ef | grep qemu | cut -c -105
<span class="line-numbers">11</span>root      42025      1 18 Nov23 ?        02:40:58 /usr/bin/qemu-system-x86_64 -name vcp-vmx1 -S -machine
<span class="line-numbers">12</span>root      42148      1 63 Nov23 ?        09:24:41 /usr/bin/qemu-system-x86_64 -name vfp-vmx1 -S -machine
<span class="line-numbers">13</span>
<span class="line-numbers">14</span>ping@trinity:~$ ps -efL | grep qemu | cut -c -105
<span class="line-numbers">15</span>root      42025      1  42025 12    4 Nov23 ?        01:47:36 /usr/bin/qemu-system-x86_64 -name vcp-vmx1
<span class="line-numbers">16</span>root      42025      1  42028  5    4 Nov23 ?        00:52:06 /usr/bin/qemu-system-x86_64 -name vcp-vmx1    <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">17</span>root      42025      1  42030  0    4 Nov23 ?        00:00:00 /usr/bin/qemu-system-x86_64 -name vcp-vmx1
<span class="line-numbers">18</span>root      42025      1  46726  0    4 11:51 ?        00:00:00 /usr/bin/qemu-system-x86_64 -name vcp-vmx1
<span class="line-numbers">19</span>
<span class="line-numbers">20</span>root      42148      1  42148  0    7 Nov23 ?        00:00:14 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">21</span>root      42148      1  42151  9    7 Nov23 ?        01:21:37 /usr/bin/qemu-system-x86_64 -name vfp-vmx1    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">22</span>root      42148      1  42152 25    7 Nov23 ?        03:42:01 /usr/bin/qemu-system-x86_64 -name vfp-vmx1    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">23</span>root      42148      1  42153 20    7 Nov23 ?        02:58:23 /usr/bin/qemu-system-x86_64 -name vfp-vmx1    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">24</span>root      42148      1  42154  9    7 Nov23 ?        01:20:30 /usr/bin/qemu-system-x86_64 -name vfp-vmx1    <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">25</span>root      42148      1  42157  0    7 Nov23 ?        00:00:00 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">26</span>root      42148      1  46722  0    7 11:51 ?        00:00:00 /usr/bin/qemu-system-x86_64 -name vfp-vmx1</code></pre>
</div>
</div>
<div class="paragraph">
<p>another capture:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:$ sudo virsh qemu-monitor-command 2 --hmp &quot;info cpus&quot;
<span class="line-numbers"> 2</span>* CPU #0: pc=0xffffffff80911183 (halted) thread_id=17760
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>ping@trinity:/virtualization/images/vmx_20151102.0/config$ sudo virsh qemu-monitor-command 3 --hmp &quot;info cpus&quot;
<span class="line-numbers"> 5</span>* CPU #0: pc=0xffffffff8100af50 (halted) thread_id=17883
<span class="line-numbers"> 6</span>  CPU #1: pc=0xffffffff8100af50 (halted) thread_id=17884
<span class="line-numbers"> 7</span>  CPU #2: pc=0xffffffff8100af50 thread_id=17885
<span class="line-numbers"> 8</span>  CPU #3: pc=0xffffffff8100af50 (halted) thread_id=17886
<span class="line-numbers"> 9</span>
<span class="line-numbers">10</span>ping@trinity:$ ps -ef | grep qemu | cut -c -105
<span class="line-numbers">11</span>root      17757      1 17 19:01 ?        00:12:14 /usr/bin/qemu-system-x86_64 -name vcp-vmx1 -S -machine
<span class="line-numbers">12</span>root      17880      1 62 19:01 ?        00:42:29 /usr/bin/qemu-system-x86_64 -name vfp-vmx1 -S -machine
<span class="line-numbers">13</span>
<span class="line-numbers">14</span>ping@trinity:$ ps -efL | grep qemu | cut -c -105
<span class="line-numbers">15</span>root      17757      1  17757 11    3 19:01 ?        00:07:58 /usr/bin/qemu-system-x86_64 -name vcp-vmx1
<span class="line-numbers">16</span>root      17757      1  17760  6    3 19:01 ?        00:04:08 /usr/bin/qemu-system-x86_64 -name vcp-vmx1
<span class="line-numbers">17</span>root      17757      1  17762  0    3 19:01 ?        00:00:00 /usr/bin/qemu-system-x86_64 -name vcp-vmx1
<span class="line-numbers">18</span>root      17880      1  17880  0    6 19:01 ?        00:00:11 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">19</span>root      17880      1  17883  9    6 19:01 ?        00:06:37 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">20</span>root      17880      1  17884 24    6 19:01 ?        00:17:01 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">21</span>root      17880      1  17885 19    6 19:01 ?        00:13:29 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">22</span>root      17880      1  17886  7    6 19:01 ?        00:05:14 /usr/bin/qemu-system-x86_64 -name vfp-vmx1
<span class="line-numbers">23</span>root      17880      1  17889  0    6 19:01 ?        00:00:00 /usr/bin/qemu-system-x86_64 -name vfp-vmx1</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>RE VCPU thread</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>PFE VCPU thread</td>
</tr>
</table>
</div>
</div>
</div>
</li>
</ul>
</div>
<div class="sidebarblock">
<div class="content">
<div class="title">Unix <code>thread</code> vs <code>process</code></div>
<div class="paragraph">
<p>a *nix Thread, is a "flow of execution" of a process, from the "coding"
perspecive it&#8217;s nothing but (almost) just a "procedure" that runs independently
from its main program. Internally from the OS perspective, A thread consists of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Program counter</p>
</li>
<li>
<p>Register set</p>
</li>
<li>
<p>Stack space</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>So if a main program contains a number of procedures. all of these procedures
are able to be scheduled to run simultaneously and/or independently by the
operating system.</p>
</div>
<div class="paragraph">
<p>A thread is running inside of a process, and differs with the concept of a
"process" in the sense that thread shares with its peer threads its:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Code segment</p>
</li>
<li>
<p>Data segment</p>
</li>
<li>
<p>Operating-system resources</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>e.g., when one thread alters a code segment memory item, all other threads see
that, and a file open with one thread is available to others. this is different
with a "process" in that each process has its own running space and resources.
communicating between processes (called "IPC") is more "expensive" and requires
some special OS utilities.</p>
</div>
<div class="paragraph">
<p>In the context of VMX, "worker thread", and "IO thread" are examples of
threads running in the host OS.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_memory"><a class="anchor" href="#_memory"></a>13. memory</h2>
<div class="sectionbody">
<div class="paragraph">
<p>memory allocated in <code>vmx.conf</code> is 16G:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>#vPFE VM parameters
FORWARDING_PLANE:
    memory-mb   : 16384
    vcpus       : 4                     <i class="conum" data-value="1"></i><b>(1)</b></pre>
</div>
</div>
<div class="paragraph">
<p>memory available in guest VM:</p>
</div>
<div class="listingblock">
<div class="title">vPFE VM:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>root@localhost:~# free -m
<span class="line-numbers">2</span>             total       used       free     shared    buffers     cached
<span class="line-numbers">3</span>Mem:         15297      11380       3917          0         13       2697
<span class="line-numbers">4</span>-/+ buffers/cache:       8669       6628
<span class="line-numbers">5</span>Swap:            0          0          0</code></pre>
</div>
</div>
<div class="paragraph">
<p>this looks a little bit smaller than expected: <code>16384-15297=1087M</code>.</p>
</div>
<div class="paragraph">
<p>So about 1G memory was "missing".</p>
</div>
<div class="paragraph">
<p>By looking at kernel boot log we will find more clues about what really
happened:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>root@localhost:~# dmesg -T | grep -i memory
<span class="line-numbers"> 2</span>[Sun Dec 20 23:59:58 2015] Scanning 1 areas for low memory corruption
<span class="line-numbers"> 3</span>[Sun Dec 20 23:59:58 2015] Base memory trampoline at [ffff880000099000] 99000 size 24576
<span class="line-numbers"> 4</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x00000000-0x000fffff]
<span class="line-numbers"> 5</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x410600000-0x4107fffff]
<span class="line-numbers"> 6</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x410000000-0x4105fffff]
<span class="line-numbers"> 7</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x400000000-0x40fffffff]
<span class="line-numbers"> 8</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x00100000-0xbfffbfff]
<span class="line-numbers"> 9</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x100000000-0x3ffffffff]
<span class="line-numbers">10</span>[Sun Dec 20 23:59:58 2015] init_memory_mapping: [mem 0x410800000-0x4108fffff]
<span class="line-numbers">11</span>[Sun Dec 20 23:59:58 2015] Early memory node ranges
<span class="line-numbers">12</span>[Sun Dec 20 23:59:58 2015] Memory: 15660932k/<strong>17048576k</strong> available (8700k kernel code, 1048984k absent, 338660k reserved, 6548k data, 1172k init)
<span class="line-numbers">13</span>[Sun Dec 20 23:59:58 2015] please try 'cgroup_disable=memory' option if you don't want memory cgroups
<span class="line-numbers">14</span>[Sun Dec 20 23:59:58 2015] Initializing cgroup subsys memory
<span class="line-numbers">15</span>[Sun Dec 20 23:59:58 2015] Freeing initrd memory: 728k freed
<span class="line-numbers">16</span>[Sun Dec 20 23:59:58 2015] Scanning for low memory corruption every 60 seconds
<span class="line-numbers">17</span>[Sun Dec 20 23:59:59 2015] Freeing unused kernel memory: 1172k freed
<span class="line-numbers">18</span>[Sun Dec 20 23:59:59 2015] Freeing unused kernel memory: 1528k freed
<span class="line-numbers">19</span>[Sun Dec 20 23:59:59 2015] Freeing unused kernel memory: 584k freed</code></pre>
</div>
</div>
<div class="paragraph">
<p>So Kernel detected <code>17048576k</code> (<code>=16649M</code>), which is much closer with what was
allocated. The rest of memory is what kernel reserved for internal tasks.</p>
</div>
<div class="paragraph">
<p><code>/proc/meminfo</code> reveals more details about memory usage.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>root@localhost:~# cat /proc/meminfo
<span class="line-numbers"> 2</span>MemTotal:       15664972 kB
<span class="line-numbers"> 3</span>MemFree:         4010928 kB
<span class="line-numbers"> 4</span>Buffers:           13956 kB
<span class="line-numbers"> 5</span>Cached:          2762276 kB
<span class="line-numbers"> 6</span>SwapCached:            0 kB
<span class="line-numbers"> 7</span>Active:           148364 kB
<span class="line-numbers"> 8</span>Inactive:        2723568 kB
<span class="line-numbers"> 9</span>Active(anon):      86864 kB
<span class="line-numbers">10</span>Inactive(anon):  2665792 kB
<span class="line-numbers">11</span>Active(file):      61500 kB
<span class="line-numbers">12</span>Inactive(file):    57776 kB
<span class="line-numbers">13</span>Unevictable:           0 kB
<span class="line-numbers">14</span>Mlocked:               0 kB
<span class="line-numbers">15</span>SwapTotal:             0 kB
<span class="line-numbers">16</span>SwapFree:              0 kB
<span class="line-numbers">17</span>Dirty:                 0 kB
<span class="line-numbers">18</span>Writeback:             0 kB
<span class="line-numbers">19</span>AnonPages:        112212 kB
<span class="line-numbers">20</span>Mapped:           177764 kB
<span class="line-numbers">21</span>Shmem:           2640572 kB
<span class="line-numbers">22</span>Slab:              70948 kB
<span class="line-numbers">23</span>SReclaimable:      30708 kB
<span class="line-numbers">24</span>SUnreclaim:        40240 kB
<span class="line-numbers">25</span>KernelStack:        1968 kB
<span class="line-numbers">26</span>PageTables:         2712 kB
<span class="line-numbers">27</span>NFS_Unstable:          0 kB
<span class="line-numbers">28</span>Bounce:                0 kB
<span class="line-numbers">29</span>WritebackTmp:          0 kB
<span class="line-numbers">30</span>CommitLimit:     3638180 kB
<span class="line-numbers">31</span>Committed_AS:    3430712 kB
<span class="line-numbers">32</span>VmallocTotal:   34359738367 kB
<span class="line-numbers">33</span>VmallocUsed:       51268 kB
<span class="line-numbers">34</span>VmallocChunk:   34359683660 kB
<span class="line-numbers">35</span>HugePages_Total:    4096
<span class="line-numbers">36</span>HugePages_Free:        0
<span class="line-numbers">37</span>HugePages_Rsvd:        0
<span class="line-numbers">38</span>HugePages_Surp:        0
<span class="line-numbers">39</span>Hugepagesize:       2048 kB
<span class="line-numbers">40</span>DirectMap4k:       13296 kB
<span class="line-numbers">41</span>DirectMap2M:     2355200 kB
<span class="line-numbers">42</span>DirectMap1G:    13631488 kB</code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_virtio_bridging"><a class="anchor" href="#_virtio_bridging"></a>14. virtio bridging</h2>
<div class="sectionbody">
<div class="paragraph">
<p>virtio provides emulations of NIC , disk and other IO devices with high
performance, but it does not has any "build-in" bridging facilities or anything
in that purpose. So in order to "bind" the virtio interfaces to the target
physical NIC, we usually need to add that "bridging" into our virtio VMX setup
after the VMs were brought up and running. This can be done with one of the
existing utilities that is available, depending on whichever is suitable for
the needs.</p>
</div>
<div class="paragraph">
<p>some popular examples of these utilities will be:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>legacy linux <code>bridge</code>,</p>
</li>
<li>
<p>open <code>vSwitch</code> (OVS),</p>
</li>
<li>
<p>Juniper <code>vRouter</code>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Here is a simplest example of "binding" 2 virtio interfaces to each other, with
linux bridge.</p>
</div>
<div class="listingblock">
<div class="title">before "binding"</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ brctl show
<span class="line-numbers"> 2</span>bridge name     bridge id               STP enabled     interfaces
<span class="line-numbers"> 3</span>br-ext          8000.38eaa7377c54       yes             br-ext-nic
<span class="line-numbers"> 4</span>                                                        em1
<span class="line-numbers"> 5</span>                                                        vcp_ext-vmx1
<span class="line-numbers"> 6</span>                                                        vfp_ext-vmx1
<span class="line-numbers"> 7</span>br-int-vmx1             8000.5254006b156e       yes     br-int-vmx1-nic
<span class="line-numbers"> 8</span>                                                        vcp_int-vmx1
<span class="line-numbers"> 9</span>                                                        vfp_int-vmx1
<span class="line-numbers">10</span>virbr0          8000.fe0417010201       yes             ge-0.0.0-vmx1
<span class="line-numbers">11</span>                                                        ge-0.0.1-vmx1</code></pre>
</div>
</div>
<div class="paragraph">
<p>"binding" 2 virtio interfaces is like to connect a "loopback cable" between
them , say , <code>ge-0/0/0</code> and <code>ge-0/0/1</code>. Changing the <code>config/vmx-junosdev.conf</code>
file so it looks:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/virtualization/images/vmx_20151102.0/config$ vim vmx-junosdev.conf.1
<span class="line-numbers"> 2</span>##############################################################
<span class="line-numbers"> 3</span>#
<span class="line-numbers"> 4</span>#  vmx-junos-dev.conf
<span class="line-numbers"> 5</span>#  - Config file for junos device bindings.
<span class="line-numbers"> 6</span>#  - Uses YAML syntax.
<span class="line-numbers"> 7</span>#  - Leave a space after &quot;:&quot; to specify the parameter value.
<span class="line-numbers"> 8</span>#  - For physical NIC, set the 'type' as 'host_dev'
<span class="line-numbers"> 9</span>#  - For junos devices, set the 'type' as 'junos_dev' and
<span class="line-numbers">10</span>#    set the mandatory parameter 'vm-name' to the name of
<span class="line-numbers">11</span>#    the vPFE where the device exists
<span class="line-numbers">12</span>#  - For bridge devices, set the 'type' as 'bridge_dev'
<span class="line-numbers">13</span>#
<span class="line-numbers">14</span>##############################################################
<span class="line-numbers">15</span>interfaces :
<span class="line-numbers">16</span>
<span class="line-numbers">17</span>     - link_name  : vmx_link4
<span class="line-numbers">18</span>       endpoint_1 :
<span class="line-numbers">19</span>         - type        : junos_dev
<span class="line-numbers">20</span>           vm_name     : vmx1
<span class="line-numbers">21</span>           dev_name    : ge-0/0/0
<span class="line-numbers">22</span>       endpoint_2 :
<span class="line-numbers">23</span>         - type        : junos_dev
<span class="line-numbers">24</span>           vm_name     : vmx1
<span class="line-numbers">25</span>           dev_name    : ge-0/0/1</code></pre>
</div>
</div>
<div class="paragraph">
<p>This read as: "please create a linux bridge <code>vmx_link4</code>, and use it to bind
junos device interface <code>ge-0/0/0</code> of VM vmx1, to the other juno device
interface <code>ge-0/0/1</code> in the same VM"</p>
</div>
<div class="paragraph">
<p>Now run the vmx.sh script the execute this action:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>sudo ./vmx.sh -lvf --bind-dev --cfg config/vmx-junosdev.conf.1
<span class="line-numbers">2</span>Checking package ethtool..........................[OK]
<span class="line-numbers">3</span>Bind Link vmx_link4(ge-0.0.0-vmx1, ge-0.0.1-vmx1)
<span class="line-numbers">4</span>[OK]</code></pre>
</div>
</div>
<div class="paragraph">
<p>As expected, this will create a new bridge named <code>vmx_link4</code> and add the JUNOS
interfaces (<code>ge-0/0/0</code> and <code>ge-0/0/1</code>) mapped corresponding virtio tap
interface (<code>ge-0.0.0-vmx1</code> and <code>ge-0.0.1-vmx1</code>) into the bridge.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/virtualization/images/vmx_20151102.0$ brctl show
<span class="line-numbers"> 2</span>bridge name     bridge id               STP enabled     interfaces
<span class="line-numbers"> 3</span>br-ext          8000.38eaa7377c54       yes             br-ext-nic
<span class="line-numbers"> 4</span>                                                        em1
<span class="line-numbers"> 5</span>                                                        vcp_ext-vmx1
<span class="line-numbers"> 6</span>                                                        vfp_ext-vmx1
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>br-int-vmx1             8000.525400fa6e56       yes     br-int-vmx1-nic
<span class="line-numbers"> 9</span>                                                        vcp_int-vmx1
<span class="line-numbers">10</span>                                                        vfp_int-vmx1
<span class="line-numbers">11</span>
<span class="line-numbers">12</span>virbr0          8000.000000000000       yes                            <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">13</span>
<span class="line-numbers">14</span>vmx_link4               8000.fe0417010201       no      ge-0.0.0-vmx1  <i class="conum" data-value="2"></i><b>(2)</b>
<span class="line-numbers">15</span>                                                        ge-0.0.1-vmx1</code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>the links were removed from the default <code>virbr0</code> bridge</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>the links were added to new bridge <code>vmx_link4</code>.</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="_build_a_setup_with_internal_connections"><a class="anchor" href="#_build_a_setup_with_internal_connections"></a>14.1. build a setup with internal connections</h3>
<div class="paragraph">
<p>now we can setup a Junos logical router based virtual test environment like
below:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>192.168.122.10            192.168.122.3
        .....vmx_link4........
        .                    .
        .                    .
   +----.--------------------.-----+
   | ge-0.0.0-vmx1  ge-0.0.1-vmx1  |
   |    .                    .     |
   |    .                    .     |
   | +--.-------+    +-------.--+  |
   | |(ge-0/0/0)|    |(ge-0/0/1)|  |
   | |192.168.  |    |192.168.  |  |
   | |122.10    |    |122.3     |  |
   | |          |    |          |  |
   | |LR:default|    |LR: r1    |  |
   | +----------+    +----------+  |
   |                               |
   +-------------------------------+</pre>
</div>
</div>
<div class="paragraph">
<p>with logical routers (LR) and internal connection between the virtio
interfaces, we now have a handy virtual multi-router setup.</p>
</div>
<div class="listingblock">
<div class="title">ping test:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>[edit]
<span class="line-numbers"> 2</span>root# run show interfaces routing
<span class="line-numbers"> 3</span>Interface        State Addresses
<span class="line-numbers"> 4</span>ge-0/0/0.0       Up    INET  192.168.122.10         #&lt;------
<span class="line-numbers"> 5</span>pfe-0/0/0.16383  Up
<span class="line-numbers"> 6</span>pfh-0/0/0.16384  Up
<span class="line-numbers"> 7</span>pfh-0/0/0.16383  Up
<span class="line-numbers"> 8</span>lc-0/0/0.32769   Up
<span class="line-numbers"> 9</span>lo0.16385        Up
<span class="line-numbers">10</span>lo0.16384        Up    INET  127.0.0.1
<span class="line-numbers">11</span>fxp0.0           Up    INET  192.168.1.105
<span class="line-numbers">12</span>em1.0            Up    INET  10.0.0.4
<span class="line-numbers">13</span>                       INET  128.0.0.1
<span class="line-numbers">14</span>                       INET  128.0.0.4
<span class="line-numbers">15</span>                       INET6 fe80::5254:ff:fe5b:901e
<span class="line-numbers">16</span>                       INET6 fec0::a:0:0:4
<span class="line-numbers">17</span>
<span class="line-numbers">18</span>[edit]
<span class="line-numbers">19</span>root# run show interfaces routing logical-system r1
<span class="line-numbers">20</span>Interface        State Addresses
<span class="line-numbers">21</span>ge-0/0/1.0       Up    INET  192.168.122.3          #&lt;------
<span class="line-numbers">22</span>
<span class="line-numbers">23</span>[edit]
<span class="line-numbers">24</span>root# run ping 192.168.122.3
<span class="line-numbers">25</span>PING 192.168.122.3 (192.168.122.3): 56 data bytes
<span class="line-numbers">26</span>64 bytes from 192.168.122.3: icmp_seq=0 ttl=63 time=15.073 ms
<span class="line-numbers">27</span>^C
<span class="line-numbers">28</span>--- 192.168.122.3 ping statistics ---
<span class="line-numbers">29</span>1 packets transmitted, 1 packets received, 0% packet loss
<span class="line-numbers">30</span>round-trip min/avg/max/stddev = 15.073/15.073/15.073/nan ms
<span class="line-numbers">31</span>
<span class="line-numbers">32</span>[edit]
<span class="line-numbers">33</span>root# run ping 192.168.122.10 logical-system r1
<span class="line-numbers">34</span>PING 192.168.122.10 (192.168.122.10): 56 data bytes
<span class="line-numbers">35</span>64 bytes from 192.168.122.10: icmp_seq=0 ttl=64 time=2.058 ms
<span class="line-numbers">36</span>^C
<span class="line-numbers">37</span>--- 192.168.122.10 ping statistics ---
<span class="line-numbers">38</span>1 packets transmitted, 1 packets received, 0% packet loss
<span class="line-numbers">39</span>round-trip min/avg/max/stddev = 2.058/2.058/2.058/0.000 ms</code></pre>
</div>
</div>
</div>
</div>
</div>
<h1 id="_part_3_vmx_kvm_features" class="sect0"><a class="anchor" href="#_part_3_vmx_kvm_features"></a>Part 3: VMX/KVM features</h1>
<div class="openblock partintro">
<div class="content">
in this part I&#8217;ll explore some of the common, and key VMX-involved KVM
features. For each topic, I&#8217;ll start with the "basic concept", then proceed
with command illustrations.
</div>
</div>
<div class="sect1">
<h2 id="_vm_management_virsh"><a class="anchor" href="#_vm_management_virsh"></a>15. VM management - virsh</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_virsh_basic_concept"><a class="anchor" href="#_virsh_basic_concept"></a>15.1. virsh basic concept</h3>
<div class="paragraph">
<p><code>Libvirt</code> is collection of open source API, daemon and management tool that
provides a convenient way to manage virtual machines and other virtualization
functionality, such as storage and network interface management. The software
components include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>an API library,</p>
</li>
<li>
<p>a daemon (libvirtd), and</p>
</li>
<li>
<p>a command line utility (virsh).</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It can be used to manage KVM, Xen, VMware ESX, QEMU and a lot of other (may
all?) currently popular virtualization technologies.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12133763/4bdf873e-b3f6-11e5-8e9f-c8483b38fb87.png" alt="4bdf873e b3f6 11e5 8e9f c8483b38fb87">
</div>
<div class="title">Figure 5. libvirt</div>
</div>
<div class="paragraph">
<p>A primary goal of libvirt is to provide a single (and simple) way to manage
multiple different virtualization providers/hypervisors - same command can be
used to manage the existing virtual machines for any supported hypervisor (KVM,
Xen, VMWare ESX, etc.)</p>
</div>
<div class="paragraph">
<p>For example: To list current running guest VMs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh list
[sudo] password for ping:
 Id    Name                           State
----------------------------------------------------
 2     vcp-vmx1                       running
 3     vfp-vmx1                       running</pre>
</div>
</div>
<div class="paragraph">
<p>The above command will list any running VMs that are managed by <code>libvirt</code>,
regardless of which hypervisor is currently in use. In this doc we&#8217;ll focus on
VMX, which is currently implemented based on <code>qemu-kvm</code> hypervisor.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
use <code>--all</code> option to list all guest VMs including those not currently
running.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p><code>Libvirt</code> also provides extensive tools and commands to manage the <code>domain</code>,
<code>node</code> and <code>network</code>.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
In <code>libvirt</code> terminology, <code>domain</code> indicates guest VM and <code>node</code> refers
to the host machine.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_libvirt_virsh_commonly_used_commands"><a class="anchor" href="#_libvirt_virsh_commonly_used_commands"></a>15.2. libvirt/virsh commonly used commands</h3>
<div class="paragraph">
<p>In this section we&#8217;ll demonstrate some of most commonly used virsh commands to
manage the host and guest OS. refer to virsh online help (<code>virsh help</code>) or
libvirt website for more details of the usage.</p>
</div>
<div class="sect3">
<h4 id="_domain_managment"><a class="anchor" href="#_domain_managment"></a>15.2.1. domain managment</h4>
<div class="listingblock">
<div class="title">dominfo: (virtual) CPU/Memory info of each domain(VM):</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh dominfo 2
<span class="line-numbers"> 2</span>Id:             2
<span class="line-numbers"> 3</span>Name:           vcp-vmx1
<span class="line-numbers"> 4</span>UUID:           90520384-41af-4029-9523-040ec59bb7f2
<span class="line-numbers"> 5</span>OS Type:        hvm
<span class="line-numbers"> 6</span>State:          running
<span class="line-numbers"> 7</span>CPU(s):         1       #&lt;------
<span class="line-numbers"> 8</span>CPU time:       7461.9s
<span class="line-numbers"> 9</span>Max memory:     2000896 KiB     #&lt;------
<span class="line-numbers">10</span>Used memory:    2000000 KiB     #&lt;------
<span class="line-numbers">11</span>Persistent:     yes
<span class="line-numbers">12</span>Autostart:      disable
<span class="line-numbers">13</span>Managed save:   no
<span class="line-numbers">14</span>Security model: none
<span class="line-numbers">15</span>Security DOI:   0
<span class="line-numbers">16</span>
<span class="line-numbers">17</span>ping@trinity:~$ sudo virsh dominfo 3
<span class="line-numbers">18</span>Id:             3
<span class="line-numbers">19</span>Name:           vfp-vmx1
<span class="line-numbers">20</span>UUID:           34991bba-ecbf-4680-905f-dc1d6b1c2308
<span class="line-numbers">21</span>OS Type:        hvm
<span class="line-numbers">22</span>State:          running
<span class="line-numbers">23</span>CPU(s):         4       #&lt;------
<span class="line-numbers">24</span>CPU time:       29660.4s
<span class="line-numbers">25</span>Max memory:     16000000 KiB    #&lt;------
<span class="line-numbers">26</span>Used memory:    16000000 KiB    #&lt;------
<span class="line-numbers">27</span>Persistent:     yes
<span class="line-numbers">28</span>Autostart:      disable
<span class="line-numbers">29</span>Managed save:   no
<span class="line-numbers">30</span>Security model: none
<span class="line-numbers">31</span>Security DOI:   0</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">vcpuinfo: state of each virtual cpu assigned to guest VM</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh vcpuinfo vcp-vmx1
<span class="line-numbers"> 2</span>VCPU:           0
<span class="line-numbers"> 3</span>CPU:            7
<span class="line-numbers"> 4</span>State:          running
<span class="line-numbers"> 5</span>CPU time:       465.7s
<span class="line-numbers"> 6</span>CPU Affinity:   -------y------------------------
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>ping@trinity:~$ sudo virsh vcpuinfo vfp-vmx1
<span class="line-numbers"> 9</span>VCPU:           0
<span class="line-numbers">10</span>CPU:            0
<span class="line-numbers">11</span>State:          running
<span class="line-numbers">12</span>CPU time:       571.7s
<span class="line-numbers">13</span>CPU Affinity:   y-------------------------------
<span class="line-numbers">14</span>
<span class="line-numbers">15</span>VCPU:           1
<span class="line-numbers">16</span>CPU:            1
<span class="line-numbers">17</span>State:          running
<span class="line-numbers">18</span>CPU time:       1432.5s
<span class="line-numbers">19</span>CPU Affinity:   -y------------------------------
<span class="line-numbers">20</span>
<span class="line-numbers">21</span>VCPU:           2
<span class="line-numbers">22</span>CPU:            2
<span class="line-numbers">23</span>State:          running
<span class="line-numbers">24</span>CPU time:       1133.6s
<span class="line-numbers">25</span>CPU Affinity:   --y-----------------------------
<span class="line-numbers">26</span>
<span class="line-numbers">27</span>VCPU:           3
<span class="line-numbers">28</span>CPU:            3
<span class="line-numbers">29</span>State:          running
<span class="line-numbers">30</span>CPU time:       517.1s
<span class="line-numbers">31</span>CPU Affinity:   ---y----------------------------</code></pre>
</div>
</div>
<div class="paragraph">
<p>the "CPU affinity" indicates the "binding" between vcpu in guest VM and
physical cpu core in host machine <sup class="footnote">[<a id="_footnoteref_5" class="footnote" href="#_footnote_5" title="View footnote.">5</a>]</sup>. This can be archived by using <code>vcpupin</code>
command below.</p>
</div>
<div class="listingblock">
<div class="title">vcpupin: display or control the vcpu to host cpu affinity</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>virsh emulatorpin vcp-vmx1 0
<span class="line-numbers"> 2</span>virsh emulatorpin vfp-vmx1 0
<span class="line-numbers"> 3</span>
<span class="line-numbers"> 4</span>virsh vcpupin vcp-vmx1 0 7
<span class="line-numbers"> 5</span>
<span class="line-numbers"> 6</span>virsh vcpupin vfp-vmx1 0 0
<span class="line-numbers"> 7</span>virsh vcpupin vfp-vmx1 1 1
<span class="line-numbers"> 8</span>virsh vcpupin vfp-vmx1 2 2
<span class="line-numbers"> 9</span>virsh vcpupin vfp-vmx1 3 3</code></pre>
</div>
</div>
<div class="paragraph">
<p>With these commands, we can bind VCPU 0 of <code>vcp-vmx1</code> VM to core 7 of host
machine, VCPU 0 of <code>vfp-vmx1</code> VM to core 0 of host machine, so on so forth.  in
the section of <a href="#VCPU_ESSENTIAL">VCPU essential</a>, we know that each "VCPU" is
essentially just a thread running in the space of one of the two QEMU
processes, so binding a "VCPU" to a host cpu core is essentially just to
"scope" the running of a thread to a specific CPU core in the host - that is
why this feature is called "CPU affinity".</p>
</div>
<div class="paragraph">
<p>The <code>emulatorpin</code> indicate the thread of hypervisor itself.</p>
</div>
<div class="listingblock">
<div class="title">vcpucount: count of VCPU in use by a VM</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@ubuntu1404:~$ sudo virsh vcpucount contrail
<span class="line-numbers">2</span>maximum      config         2
<span class="line-numbers">3</span>maximum      live           2
<span class="line-numbers">4</span>current      config         2
<span class="line-numbers">5</span>current      live           2</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">storage/images of each VM:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>    ping@trinity:~$ sudo virsh domblklist 2
<span class="line-numbers">2</span>    Target     Source
<span class="line-numbers">3</span>    ------------------------------------------------
<span class="line-numbers">4</span>    hda        /virtualization/images/vmx_20151102.0/build/vmx1/images/jinstall64-vmx-15.1F-20151104.0-domestic.img
<span class="line-numbers">5</span>    hdb        /virtualization/images/vmx_20151102.0/build/vmx1/images/vmxhdd.img
<span class="line-numbers">6</span>    sda        /virtualization/images/vmx_20151102.0/images/metadata_usb.img</code></pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh domblklist 3
Target     Source
------------------------------------------------
hda        /virtualization/images/vmx_20151102.0/build/vmx1/images/vFPC-20151102.img</pre>
</div>
</div>
<div class="paragraph">
<p>This command is pretty handy that with it we can quickly locate from which
images the current VMs were built from, and where are the image files currently
located.</p>
</div>
<div class="listingblock">
<div class="title">useful qemu-monitor-command: this can be used to print the <code>qmp</code> command</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh qemu-monitor-command vcp-vmx1 --hmp "info network"
net0: index=0,type=nic,model=e1000,macaddr=02:04:17:01:01:01
 \ hostnet0: index=0,type=tap,fd=18
net1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:8d:45:1e
 \ hostnet1: index=0,type=tap,fd=19

ping@trinity:~$ sudo virsh qemu-monitor-command vfp-vmx1 --hmp "info network"
net0: index=0,type=nic,model=virtio-net-pci,macaddr=02:04:17:01:01:02
 \ hostnet0: index=0,type=tap,fd=18
net1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:5e:96:98
 \ hostnet1: index=0,type=tap,fd=21

ping@trinity:~$ sudo virsh qemu-monitor-command 2 --hmp "info cpus"
* CPU #0: pc=0xffffffff80921b83 thread_id=42028

ping@trinity:~$ sudo virsh qemu-monitor-command 3 --hmp "info cpus"
* CPU #0: pc=0xffffffff8100af50 (halted) thread_id=42151
  CPU #1: pc=0xffffffff8100af50 (halted) thread_id=42152
  CPU #2: pc=0xffffffff8100af50 (halted) thread_id=42153
  CPU #3: pc=0xffffffff8100af50 (halted) thread_id=42154</pre>
</div>
</div>
<div class="paragraph">
<p>This is sometimes quite useful because it removes the need to acquire a <code>qmp</code>
console of a guest VM in order to send <code>qmp</code> command.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
The QEMU Machine Protocol (QMP) is a JSON-based protocol which allows
applications to control a QEMU instance. in another word you can query or
manipulate the running status of a VM ,without even "login" into the VM itself.
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_node_management"><a class="anchor" href="#_node_management"></a>15.2.2. node management</h4>
<div class="paragraph">
<p>"node" means host machine under the context of <code>virsh</code>. So all commands listed
here is about the host, not the VM - This effectively provides another
<code>unified</code> CLI to manage the host machine, regardless of the OS types.
Considering there are so many OS variations and each may come with different ,
aften incompatible CLIes, using <code>virsh</code> may be easier to identify some common
data about the host.</p>
</div>
<div class="listingblock">
<div class="title">nodeinfo: cpu+memory brief overview</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh nodeinfo
CPU model:           x86_64
CPU(s):              32
CPU frequency:       3300 MHz
CPU socket(s):       1
Core(s) per socket:  8
Thread(s) per core:  1
NUMA cell(s):        4
Memory size:         528417400 KiB

labroot@MX86-host-BL660C-B1:~$ sudo virsh nodeinfo
CPU model:           x86_64
CPU(s):              32
CPU frequency:       1200 MHz
CPU socket(s):       1
Core(s) per socket:  8
Thread(s) per core:  1
NUMA cell(s):        4
Memory size:         528417400 KiB</pre>
</div>
</div>
<div class="listingblock">
<div class="title">sysinfo: host system general info in more detail (bios/cpu/memory/etc)</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh sysinfo
&lt;sysinfo type='smbios'&gt;
  &lt;bios&gt;        #&lt;------similar to "dmidecode -s bios-version"
    &lt;entry name='vendor'&gt;HP&lt;/entry&gt;
    &lt;entry name='version'&gt;I32&lt;/entry&gt;
    &lt;entry name='date'&gt;02/10/2014&lt;/entry&gt;
  &lt;/bios&gt;
  &lt;system&gt;      #&lt;------similar to "dmidecode -t 1"
    &lt;entry name='manufacturer'&gt;HP&lt;/entry&gt;
    &lt;entry name='product'&gt;ProLiant BL660c Gen8&lt;/entry&gt;
    &lt;entry name='version'&gt;Not Specified&lt;/entry&gt;
    &lt;entry name='serial'&gt;USE4379WSS      &lt;/entry&gt;
    &lt;entry name='uuid'&gt;31393736-3831-5355-4534-333739575353&lt;/entry&gt;
    &lt;entry name='sku'&gt;679118-B21      &lt;/entry&gt;
    &lt;entry name='family'&gt;ProLiant&lt;/entry&gt;
  &lt;/system&gt;
  &lt;processor&gt;   #&lt;------CPU socket: similar to "dmidecode -t 4"
    &lt;entry name='socket_destination'&gt;Proc 1&lt;/entry&gt;
    &lt;entry name='type'&gt;Central Processor&lt;/entry&gt;
    &lt;entry name='family'&gt;Xeon&lt;/entry&gt;
    &lt;entry name='manufacturer'&gt;Intel&lt;/entry&gt;
    &lt;entry name='signature'&gt;Type 0, Family 6, Model 62, Stepping 4&lt;/entry&gt;
    &lt;entry name='version'&gt; Intel(R) Xeon(R) CPU E5-4627 v2 @ 3.30GHz&lt;/entry&gt;
    &lt;entry name='external_clock'&gt;100 MHz&lt;/entry&gt;
    &lt;entry name='max_speed'&gt;4800 MHz&lt;/entry&gt;
    &lt;entry name='status'&gt;Populated, Enabled&lt;/entry&gt;
    &lt;entry name='serial_number'&gt;Not Specified&lt;/entry&gt;
    &lt;entry name='part_number'&gt;Not Specified&lt;/entry&gt;
  &lt;/processor&gt;
  &lt;processor&gt;
    ......
  &lt;/processor&gt;
  &lt;memory_device&gt; #&lt;------similar to "dmidecode -t 17"
    &lt;entry name='size'&gt;32 GB&lt;/entry&gt;
    &lt;entry name='form_factor'&gt;DIMM&lt;/entry&gt;
    &lt;entry name='locator'&gt;PROC  1 DIMM  1&lt;/entry&gt;
    &lt;entry name='bank_locator'&gt;Not Specified&lt;/entry&gt;
    &lt;entry name='type'&gt;DDR3&lt;/entry&gt;
    &lt;entry name='type_detail'&gt;Synchronous&lt;/entry&gt;
    &lt;entry name='speed'&gt;1866 MHz&lt;/entry&gt;
    &lt;entry name='manufacturer'&gt;HP&lt;/entry&gt;
    &lt;entry name='serial_number'&gt;Not Specified&lt;/entry&gt;
    &lt;entry name='part_number'&gt;712384-081&lt;/entry&gt;
  &lt;/memory_device&gt;
  &lt;memory_device&gt;
  ....
  &lt;/memory_device&gt;
  ....

&lt;/sysinfo&gt;</pre>
</div>
</div>
<div class="literalblock">
<div class="title">freecell: available memory</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh freecell
Total: 520897088 KiB</pre>
</div>
</div>
<div class="listingblock">
<div class="title">nodememstats</div>
<div class="content">
<pre>ping@ubuntu1404:~$ sudo virsh nodememstats
total  :              8033536 KiB
free   :              2874116 KiB
buffers:              1333740 KiB
cached :              1462096 KiB</pre>
</div>
</div>
<div id="VIRSH_CAP" class="listingblock">
<div class="title">capabilities: system supported features</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh capabilities
&lt;capabilities&gt;

  &lt;host&gt;
    &lt;uuid&gt;36373931-3138-5553-4534-333739575353&lt;/uuid&gt;
    &lt;cpu&gt;       #&lt;------cpu flags: "cat /proc/cpuinfo"
      &lt;arch&gt;x86_64&lt;/arch&gt;
      &lt;model&gt;SandyBridge&lt;/model&gt;
      &lt;vendor&gt;Intel&lt;/vendor&gt;
      &lt;topology sockets='1' cores='8' threads='1'/&gt;
      &lt;feature name='invtsc'/&gt;
      &lt;feature name='erms'/&gt;
      &lt;feature name='smep'/&gt;
      &lt;feature name='fsgsbase'/&gt;
      &lt;feature name='pdpe1gb'/&gt;
      &lt;feature name='rdrand'/&gt;
      &lt;feature name='f16c'/&gt;
      &lt;feature name='osxsave'/&gt;
      &lt;feature name='dca'/&gt;
      &lt;feature name='pcid'/&gt;
      &lt;feature name='pdcm'/&gt;
      &lt;feature name='xtpr'/&gt;
      &lt;feature name='tm2'/&gt;
      &lt;feature name='est'/&gt;
      &lt;feature name='smx'/&gt;
      &lt;feature name='vmx'/&gt;
      &lt;feature name='ds_cpl'/&gt;
      &lt;feature name='monitor'/&gt;
      &lt;feature name='dtes64'/&gt;
      &lt;feature name='pbe'/&gt;
      &lt;feature name='tm'/&gt;
      &lt;feature name='ht'/&gt;
      &lt;feature name='ss'/&gt;
      &lt;feature name='acpi'/&gt;
      &lt;feature name='ds'/&gt;
      &lt;feature name='vme'/&gt;
      &lt;pages unit='KiB' size='4'/&gt;
      &lt;pages unit='KiB' size='2048'/&gt;
    &lt;/cpu&gt;
    &lt;power_management&gt;
      &lt;suspend_disk/&gt;
      &lt;suspend_hybrid/&gt;
    &lt;/power_management&gt;
    &lt;migration_features&gt;
      &lt;live/&gt;
      &lt;uri_transports&gt;
        &lt;uri_transport&gt;tcp&lt;/uri_transport&gt;
      &lt;/uri_transports&gt;
    &lt;/migration_features&gt;
    &lt;topology&gt;
      &lt;cells num='4'&gt;
        &lt;cell id='0'&gt;   #&lt;------NUMA node0
          &lt;memory unit='KiB'&gt;132067432&lt;/memory&gt;
          &lt;pages unit='KiB' size='4'&gt;33016858&lt;/pages&gt;
          &lt;pages unit='KiB' size='2048'&gt;0&lt;/pages&gt;
          &lt;distances&gt;
            &lt;sibling id='0' value='10'/&gt;
            &lt;sibling id='1' value='21'/&gt;
            &lt;sibling id='2' value='21'/&gt;
            &lt;sibling id='3' value='21'/&gt;
          &lt;/distances&gt;
          &lt;cpus num='8'&gt;
            &lt;cpu id='0' socket_id='0' core_id='0' siblings='0'/&gt;
            &lt;cpu id='1' socket_id='0' core_id='1' siblings='1'/&gt;
            &lt;cpu id='2' socket_id='0' core_id='2' siblings='2'/&gt;
            &lt;cpu id='3' socket_id='0' core_id='3' siblings='3'/&gt;
            &lt;cpu id='4' socket_id='0' core_id='4' siblings='4'/&gt;
            &lt;cpu id='5' socket_id='0' core_id='5' siblings='5'/&gt;
            &lt;cpu id='6' socket_id='0' core_id='6' siblings='6'/&gt;
            &lt;cpu id='7' socket_id='0' core_id='7' siblings='7'/&gt;
          &lt;/cpus&gt;
        &lt;/cell&gt;
        ......
    &lt;/topology&gt;
    &lt;secmodel&gt;
      &lt;model&gt;none&lt;/model&gt;
      &lt;doi&gt;0&lt;/doi&gt;
    &lt;/secmodel&gt;
    &lt;secmodel&gt;
      &lt;model&gt;dac&lt;/model&gt;
      &lt;doi&gt;0&lt;/doi&gt;
      &lt;baselabel type='kvm'&gt;+0:+0&lt;/baselabel&gt;
      &lt;baselabel type='qemu'&gt;+0:+0&lt;/baselabel&gt;
    &lt;/secmodel&gt;
  &lt;/host&gt;

  &lt;guest&gt;
    &lt;os_type&gt;hvm&lt;/os_type&gt;
    &lt;arch name='i686'&gt;
      &lt;wordsize&gt;32&lt;/wordsize&gt;
      &lt;emulator&gt;/usr/bin/qemu-system-i386&lt;/emulator&gt;
      &lt;machine canonical='pc-i440fx-trusty' maxCpus='255'&gt;pc&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.12&lt;/machine&gt;
        ......
      &lt;machine maxCpus='255'&gt;pc-i440fx-2.0&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.13&lt;/machine&gt;
      &lt;domain type='qemu'&gt;
      &lt;/domain&gt;
      &lt;domain type='kvm'&gt;
        &lt;emulator&gt;/usr/bin/kvm&lt;/emulator&gt;
        &lt;machine canonical='pc-i440fx-trusty' maxCpus='255'&gt;pc&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.3&lt;/machine&gt;
        ......
        &lt;machine maxCpus='255'&gt;pc-0.13&lt;/machine&gt;
      &lt;/domain&gt;
    &lt;/arch&gt;
    &lt;features&gt;
      &lt;cpuselection/&gt;
      &lt;deviceboot/&gt;
      &lt;disksnapshot default='on' toggle='no'/&gt;
      &lt;acpi default='on' toggle='yes'/&gt;
      &lt;apic default='on' toggle='no'/&gt;
      &lt;pae/&gt;
      &lt;nonpae/&gt;
    &lt;/features&gt;
  &lt;/guest&gt;

  &lt;guest&gt;
    ......
  &lt;/guest&gt;

&lt;/capabilities&gt;</pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
this output is quite long so not everything is printed here. for
completeness the whole capture is available in <a href="#appendix">appendix</a>
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_network_and_interface_management"><a class="anchor" href="#_network_and_interface_management"></a>15.2.3. network and interface management</h4>
<div class="paragraph">
<p>network and interface management virsh commands are used to manage the network
resources in the host.</p>
</div>
<div class="literalblock">
<div class="title">net-list: list active (or all if also with <code>--all</code>) virtual networks</div>
<div class="content">
<pre>labroot@MX86-host-BL660C-B1:~$ sudo virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br-ext               active     no            yes
 br-int               active     no            yes
 default              active     yes           yes</pre>
</div>
</div>
<div class="literalblock">
<div class="title">net-info: print properties of a specific network</div>
<div class="content">
<pre>labroot@MX86-host-BL660C-B1:~$ sudo virsh net-info br-ext
Name:           br-ext
UUID:           efef3d11-cf4f-4cc2-9d21-d1814c0e2e0e
Active:         yes
Persistent:     yes
Autostart:      no
Bridge:         br-ext</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>labroot@MX86-host-BL660C-B1:~$ sudo virsh net-info br-int
Name:           br-int
UUID:           e8af7e46-f574-4d8d-96ef-210a8d61067b
Active:         yes
Persistent:     yes
Autostart:      no
Bridge:         br-int</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>labroot@MX86-host-BL660C-B1:~$ sudo virsh net-info default
Name:           default
UUID:           0969babb-7b8f-457d-8548-1783152ad05d
Active:         yes
Persistent:     yes
Autostart:      yes
Bridge:         virbr0</pre>
</div>
</div>
<div class="literalblock">
<div class="title">iface-list: list all active (up) interfaces in host</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh iface-list
 Name                 State      MAC Address
---------------------------------------------------
 br-ext               active     38:ea:a7:37:7c:54
 p2p1                 active     38:ea:a7:17:65:a0
 p3p1                 active     38:ea:a7:17:65:84</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@matrix:/home$ sudo virsh iface-list
[sudo] password for ping:
 Name                 State      MAC Address
---------------------------------------------------
 br-int-vmx1          active     52:54:00:04:5f:fe
 br-int-vmx2          active     52:54:00:d8:7b:dc
 br0                  active     5c:b9:01:8a:f0:b8
 br1                  active     5c:b9:01:8a:f0:b9
 p1p1                 active     8c:dc:d4:b7:79:d0
 p1p2                 active     8c:dc:d4:b7:79:d1
 p2p1                 active     8c:dc:d4:b7:7a:e8
 p2p2                 active     8c:dc:d4:b7:7a:e9</pre>
</div>
</div>
<div class="literalblock">
<div class="title">iface-mac: print MAC address of one specific interface in host</div>
<div class="content">
<pre>ping@trinity:~$ sudo virsh iface-mac p2p1
38:ea:a7:17:65:a0</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="VT-D"><a class="anchor" href="#VT-D"></a>16. iommu/VT-d</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_vt_d_basic_concept"><a class="anchor" href="#_vt_d_basic_concept"></a>16.1. VT-d basic concept</h3>
<div class="paragraph">
<p>I/O Virtualization (IOV) involves <strong>sharing</strong> a single I/O resource between
multiple virtual machines. Approaches for IOV include (but may not be limited
to):</p>
</div>
<div class="ulist">
<ul>
<li>
<p>software based approach</p>
</li>
<li>
<p>direct assignment</p>
</li>
<li>
<p>SR-IOV</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="_software_based_approach"><a class="anchor" href="#_software_based_approach"></a>16.1.1. software based approach</h4>
<div class="paragraph">
<p>Software based sharing utilizes emulation techniques to provide a logical I/O
hardware device to the VM. The emulation layer interposes itself between the
driver running in the guest OS and the underlying hardware. This level of
indirection allows the VMM to intercept all traffic issued by the guest driver.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/13040385/60a3baa8-d379-11e5-9228-c476b575433d.png" alt="60a3baa8 d379 11e5 9228 c476b575433d">
</div>
<div class="title">Figure 6. software based I/O sharing</div>
</div>
<div class="paragraph">
<p>Emulation software involves following common tasks:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>parse the I/O commands,</p>
</li>
<li>
<p>translate guest addresses into host physical addresses</p>
</li>
<li>
<p>ensure that all referenced memory pages are present in memory.</p>
</li>
<li>
<p>resolve the multiple I/O requests from all the virtual machines</p>
</li>
<li>
<p>serialize them into a single I/O stream that can be handled by the underlying
hardware.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>There are at least 2 common approaches to software-based sharing:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>device emulation</p>
</li>
<li>
<p>the splitdriver model:</p>
</li>
</ul>
</div>
<div class="paragraph">
<div class="title">Device emulation models</div>
<p>In this method, the hypervisor mimic widely supported real devices (such as an
Intel 1Gb NIC) and utilize existing drivers in the guest OS. The VMM emulates
the I/O device to ensure compatibility and then processes I/O operations before
passing them on to the physical device (which may be different).</p>
</div>
<div class="paragraph">
<p>an example of the implementation of this model is QEMU, which can emulate a lot
of legacy devices that can be used in the VM, regardless of real physical
device type available in the host.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ qemu-system-x86_64 -device ?
...&lt;snippet&gt;...
name "ioh3420", bus PCI, desc "Intel IOH device id 3420 PCIE Root Port"
...&lt;snippet&gt;...
name "piix3-ide", bus PCI
name "piix3-ide-xen", bus PCI
...&lt;snippet&gt;...
name "e1000", bus PCI, desc "Intel Gigabit Ethernet"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
name "i82557b", bus PCI, desc "Intel i82557B Ethernet"
name "i82557c", bus PCI, desc "Intel i82557C Ethernet"
name "i82558a", bus PCI, desc "Intel i82558A Ethernet"
...&lt;snippet&gt;...
name "isa-parallel", bus ISA
name "isa-serial", bus ISA
...&lt;snippet&gt;...
name "isa-vga", bus ISA
...&lt;snippet&gt;...
name "adlib", bus ISA, desc "Yamaha YM3812 (OPL2)"</pre>
</div>
</div>
<div class="paragraph">
<p>A potential problem is that I/O operations then have to traverse two I/O
stacks, one in the VM and one in the VMM. Because of the low performance
archived, this method is rarely used in an environment where performance is
key.</p>
</div>
<div class="paragraph">
<div class="title">split-driver model</div>
<p>This method takes a similar approach but, instead of emulating a legacy device,
the split-driver uses a front-end driver in the guest that works in concert
with a back-end driver in the VMM. These drivers are optimized for sharing and
have the benefit of not needing to emulate an entire device. The back-end
driver communicates with the physical device.</p>
</div>
<div class="paragraph">
<p>An example of the implementation of this model is <code>virtio</code>, which will be
discussed in section <a href="#VIRTIO">virtio</a>.</p>
</div>
<div class="paragraph">
<p>Both the device emulation and split-driver (para-virtualized driver) provide a
subset of the total functionality provided by physical hardware and may as a
result not have the ability to take advantage of advanced capabilities provided
by the device.</p>
</div>
<div class="paragraph">
<p>In addition, significant CPU overhead may be required by the VMM to implement a
virtual software-based switch that routes packets to and from the appropriate
VMS. This CPU overhead can (and generally does) reduce the maximum throughput
on an I/O device. As an example, extensive testing has shown using only device
emulation, a 10Gbps Ethernet controller can achieve a maximum throughput of 4.5
to around 6.5 Gbps (the range varies with the architecture of the server being
tested on).</p>
</div>
<div class="paragraph">
<p>One reason that line rate, or near line rate cannot be achieved is because each
packet must go through the software switch and that requires CPU cycles to
process the packets.</p>
</div>
</div>
<div class="sect3">
<h4 id="_direct_assignment"><a class="anchor" href="#_direct_assignment"></a>16.1.2. direct assignment</h4>
<div class="paragraph">
<p>Software-based sharing adds overhead to each I/O operation due to the emulation
layer between the guest driver and the I/O hardware. This indirection has the
additional affect of eliminating the use of hardware acceleration that may be
available in the physical device.</p>
</div>
<div class="paragraph">
<p>Such problems can be reduced by directly exposing the hardware to the guest OS
and running a native device driver.</p>
</div>
<div class="paragraph">
<p>Intel has added enhancements to facilitate memory translation and ensure
protection of memory that enables a device to directly DMA to/from host memory.
These enhancements provide the ability to bypass the VMM’s I/O emulation layer
and can result in throughput improvement for the VMs.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Intel® <code>VT-x</code> : allows a VM to have direct access to a physical address (if so
configured by the VMM). This ability allows a device driver within a VM to be
able to write directly to registers IO device (such as configuring DMA
descriptors).</p>
</li>
<li>
<p>Intel® <code>VT-d</code> : provides a similar capability for IO devices to be able to
write directly to the memory space of a virtual machine, for example a DMA
operation. VT-d technology is illustrated below:</p>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/13031781/60a710dc-d2a6-11e5-98fb-49979ffbad2d.png" alt="60a710dc d2a6 11e5 98fb 49979ffbad2d">
</div>
<div class="title">Figure 7. direct assignment</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>VT-d</code> is Intel&#8217;s IOMMU implementation. It provides I/O device assignment, DMA
remapping, interrupt remapping and other features to improve the performance
and security of the virtualization environment.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>At least in the context and test environment of this doc, following
technical terms refer almost the same thing:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>IOMMU</code></p>
</li>
<li>
<p><code>VT-d</code></p>
</li>
<li>
<p><code>PCI device passthgouth</code></p>
</li>
<li>
<p><code>direct assignment</code></p>
</li>
</ul>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>In simple terms, with <code>VT-d</code> support it&#8217;s possible to "detach" a device from
host, and then attach it into a guest VM. This way the IO performance of the VM
can be sharply increased.</p>
</div>
<div class="paragraph">
<p>In my setup, VT-d was enabled in the  <a href="#IOMMU"><code>IOMMU</code></a> section as part of
<a href="#preparation">system preparation</a>.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>the installation script check <code>/boot/grub/grub.cfg</code> file to verify whether
iommu has been enabled or not.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ cat /boot/grub/grub.cfg | grep &quot;intel_iommu=on&quot;
<span class="line-numbers">2</span>        linux   /vmlinuz-3.19.0-25-generic root=/dev/mapper/trinity--vg-root ro  intel_iommu=on
<span class="line-numbers">3</span>                linux   /vmlinuz-3.19.0-25-generic root=/dev/mapper/trinity--vg-root ro  intel_iommu=on
<span class="line-numbers">4</span>                linux   /vmlinuz-3.16.0-30-generic root=/dev/mapper/trinity--vg-root ro  intel_iommu=on
<span class="line-numbers">5</span>                linux   /vmlinuz-3.13.0-32-generic root=/dev/mapper/trinity--vg-root ro  intel_iommu=on</code></pre>
</div>
</div>
</td>
</tr>
</table>
</div>
<div class="sidebarblock">
<div class="content">
<div class="ulist">
<div class="title">about VT-d, VT-x, SR-IOV virtualization features</div>
<ul>
<li>
<p>Intel <code>VT-d</code> extensions: or "d"irect-I/O, direct assignment: add
virtualization support to <strong>Intel chipsets</strong> that can "assign" specific I/O
devices to specific virtual machines (VM)s.</p>
</li>
<li>
<p>Intel <code>VT-x</code> is virtualization extension specificly on <strong><code>CPU</code></strong>, while <code>VT-d</code>
is virtualization on <strong><code>chipset</code></strong>. VT-x can be thought of as "basic
virtualization architecture" in Intel CPU.</p>
</li>
<li>
<p><code>SR-IOV</code> is mostly (but "in thoery" not limited to) virtualization on NIC
card.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Here is a brief and not-so-acurate summary of each term and its category of
virtualization:</p>
</div>
<table class="tableblock frame-all grid-all" style="width: 50%;">
<colgroup>
<col style="width: 50%;">
<col style="width: 50%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">term</th>
<th class="tableblock halign-left valign-top">virtualization category</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VT-x</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VT-d</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">chipset</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">SR-IOV</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">NIC</p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_pci_stub"><a class="anchor" href="#_pci_stub"></a>16.2. pci_stub</h3>
<div class="paragraph">
<p><code>pci_stub</code> is a kernel driver module that is used to "hide" the device from
host or any other VMs that were not assigned to use this device. This is
required to complete the VT-d/PCI-passthough operation.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ modinfo pci_stub
<span class="line-numbers"> 2</span>filename:       /lib/modules/3.13.0-32-generic/kernel/drivers/pci/pci-stub.ko
<span class="line-numbers"> 3</span>author:         Chris Wright &lt;chrisw@sous-sol.org&gt;
<span class="line-numbers"> 4</span>license:        GPL
<span class="line-numbers"> 5</span>srcversion:     194150416ED68735C4D2803
<span class="line-numbers"> 6</span>depends:
<span class="line-numbers"> 7</span>intree:         Y
<span class="line-numbers"> 8</span>vermagic:       3.13.0-32-generic SMP mod_unload modversions
<span class="line-numbers"> 9</span>signer:         Magrathea: Glacier signing key
<span class="line-numbers">10</span>sig_key:        5E:3C:0F:9C:A6:E3:65:43:53:5F:A2:BB:5B:70:9E:84:F1:6D:A7:C7
<span class="line-numbers">11</span>sig_hashalgo:   sha512
<span class="line-numbers">12</span>parm:           ids:Initial PCI IDs to add to the stub driver, format is
<span class="line-numbers">13</span>                &quot;vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]&quot; and multiple
<span class="line-numbers">14</span>                comma separated entries can be specified (string)</code></pre>
</div>
</div>
<div class="paragraph">
<p><code>pci_stub</code> was compiled as a kernel module and not to be loaded by default in
our case.  It can be loaded manually with <code>modprobe</code>.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ lsmod | grep pci_stub
<span class="line-numbers">2</span>ping@trinity:~$ modprobe pci_stub
<span class="line-numbers">3</span>ping@trinity:~$ lsmod | grep pci_stub
<span class="line-numbers">4</span>pci_stub               12622  0</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_process_to_assign_pci_device_in_kvm"><a class="anchor" href="#_process_to_assign_pci_device_in_kvm"></a>16.3. process to assign PCI device in KVM</h3>
<div class="paragraph">
<p>Here are the steps to isolate a PCI device, in this example, a SR-IOV VF:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>before assignment, <code>p3p1 VF 0</code> is with <code>ixgbevf</code> driver</p>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:/images/vmx_20151102.0/build$ sudo lspci -nvks 0000:23:10.0
23:10.0 0200: <strong>8086:10ed</strong> (rev 01)
        Subsystem: 103c:17d2
        Flags: bus master, fast devsel, latency 0
        [virtual] Memory at f3400000 (64-bit, prefetchable) [size=16K]
        [virtual] Memory at f3300000 (64-bit, prefetchable) [size=16K]
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: ixgbevf</pre>
</div>
</div>
<div class="paragraph">
<p>the <code>-n</code> option will show PCI vendor and device codes as numbers instead of
looking them up in the PCI ID list. In this output it give wendor number
<strong>8096:10ed</strong> , which we&#8217;ll need to use in later commands.</p>
</div>
</li>
<li>
<p>now "unbind" the VF from host, and "bind" it to <code>pci-stub</code> driver</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:/images/vmx_20151102.0/build$ sudo -i
<span class="line-numbers">2</span>ping@trinity:~# echo &quot;8086 10ed&quot; &gt; /sys/bus/pci/drivers/pci-stub/new_id
<span class="line-numbers">3</span>root@trinity:~# echo 0000:23:10.0 &gt; /sys/bus/pci/devices/0000:23:10.0/driver/unbind
<span class="line-numbers">4</span>root@trinity:~# echo 0000:23:10.0 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">5</span>root@trinity:~# exit</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
these operations requires root privilege to perform.
</td>
</tr>
</table>
</div>
</li>
<li>
<p>after the "unbind", the <code>p3p1 VF 0</code> device is now with pci-stub driver
instead of the ixgbevf driver, effectively get "detached" from the host.</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/images/vmx_20151102.0/build$ sudo lspci -nvks 0000:23:10.0
<span class="line-numbers"> 2</span>23:10.0 0200: 8086:10ed (rev 01)
<span class="line-numbers"> 3</span>        Subsystem: 103c:17d2
<span class="line-numbers"> 4</span>        Flags: fast devsel
<span class="line-numbers"> 5</span>        [virtual] Memory at f3400000 (64-bit, prefetchable) [size=16K]
<span class="line-numbers"> 6</span>        [virtual] Memory at f3300000 (64-bit, prefetchable) [size=16K]
<span class="line-numbers"> 7</span>        Capabilities: [70] MSI-X: Enable- Count=3 Masked-
<span class="line-numbers"> 8</span>        Capabilities: [a0] Express Endpoint, MSI 00
<span class="line-numbers"> 9</span>        Capabilities: [100] Advanced Error Reporting
<span class="line-numbers">10</span>        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
<span class="line-numbers">11</span>        Kernel driver in use: pci-stub      #&lt;------
<span class="line-numbers">12</span>
<span class="line-numbers">13</span>ping@trinity:/images/vmx_20151102.0/build$ sudo lspci -nvks 0000:06:10.0
<span class="line-numbers">14</span>06:10.0 0200: 8086:10ed (rev 01)
<span class="line-numbers">15</span>        Subsystem: 103c:17d2
<span class="line-numbers">16</span>        Flags: fast devsel
<span class="line-numbers">17</span>        [virtual] Memory at ee300000 (64-bit, prefetchable) [size=16K]
<span class="line-numbers">18</span>        [virtual] Memory at ee200000 (64-bit, prefetchable) [size=16K]
<span class="line-numbers">19</span>        Capabilities: [70] MSI-X: Enable- Count=3 Masked-
<span class="line-numbers">20</span>        Capabilities: [a0] Express Endpoint, MSI 00
<span class="line-numbers">21</span>        Capabilities: [100] Advanced Error Reporting
<span class="line-numbers">22</span>        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
<span class="line-numbers">23</span>        Kernel driver in use: pci-stub      #&lt;------</code></pre>
</div>
</div>
</li>
</ol>
</div>
<div class="paragraph">
<p>now the <code>p3p1 VF 0</code> device is ready to be used in guest VM.</p>
</div>
<div class="paragraph">
<p>The process to assign a device back from guest VM to host machine is similar -
just unbind it from current <code>pci-stub</code> driver and then bind it back to <code>ixgbevf</code>
driver will suffice:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/images/vmx_20151102.0/build$ sudo -i
<span class="line-numbers"> 2</span>root@trinity:~# echo &quot;8086 10ed&quot; &gt; /sys/bus/pci/drivers/ixgbevf/new_id
<span class="line-numbers"> 3</span>root@trinity:~# echo 0000:23:10.0 &gt; /sys/bus/pci/devices/0000:23:10.0/driver/unbind
<span class="line-numbers"> 4</span>root@trinity:~# echo 0000:23:10.0 &gt;&gt; /sys/bus/pci/drivers/ixgbevf/bind
<span class="line-numbers"> 5</span>root@trinity:~# exit
<span class="line-numbers"> 6</span>
<span class="line-numbers"> 7</span>ping@trinity:/images/vmx_20151102.0/build$ sudo lspci -nvvs 0000:23:10.0
<span class="line-numbers"> 8</span>23:10.0 0200: 8086:10ed (rev 01)
<span class="line-numbers"> 9</span>        Subsystem: 103c:17d2
<span class="line-numbers">10</span>        Flags: bus master, fast devsel, latency 0
<span class="line-numbers">11</span>        [virtual] Memory at f3400000 (64-bit, prefetchable) [size=16K]
<span class="line-numbers">12</span>        [virtual] Memory at f3300000 (64-bit, prefetchable) [size=16K]
<span class="line-numbers">13</span>        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
<span class="line-numbers">14</span>        Capabilities: [a0] Express Endpoint, MSI 00
<span class="line-numbers">15</span>        Capabilities: [100] Advanced Error Reporting
<span class="line-numbers">16</span>        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
<span class="line-numbers">17</span>        Kernel driver in use: ixgbevf       #&lt;------ back to ixgbevf</code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_sriov"><a class="anchor" href="#_sriov"></a>17. SRIOV</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_sriov_basic_concept"><a class="anchor" href="#_sriov_basic_concept"></a>17.1. SRIOV basic concept</h3>
<div class="sect3">
<h4 id="_what_is_io_virtualization"><a class="anchor" href="#_what_is_io_virtualization"></a>17.1.1. what is "IO virtualization"?</h4>
<div class="paragraph">
<p>Input/output (I/O) virtualization is a methodology to simplify management,
lower costs and improve performance of servers in enterprise environments. I/O
virtualization environments are created by abstracting the upper layer
protocols from the physical connections.</p>
</div>
<div class="paragraph">
<p>SR-IOV is a type of "IO virtualization" technique Developed by the <code>PCI-SIG</code>
(PCI Special Interest Group), so it is also called <code>PCI-SIG SR-IOV</code>.</p>
</div>
<div class="paragraph">
<p>The <a href="http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf">SR-IOV spec</a> defined a very basic SR-IOV concept: enabling a
Single "Root Function" (for example, a single Ethernet port), to appear as
multiple, separate, devices. A physical device with SR-IOV capabilities can be
configured to appear in the PCI configuration space as multiple "virtual
functions".</p>
</div>
</div>
<div class="sect3">
<h4 id="_what_is_a_function"><a class="anchor" href="#_what_is_a_function"></a>17.1.2. what is a "function"?</h4>
<div class="paragraph">
<p>traditionally, a PCIe Device has a Unique PCI Function Address, in the form of
<code>Bus</code> <code>domain</code> and <code>Function</code>, sometimes called a <a href="#BDF">BDF notation</a>, which
can be viewed by CLI command <a href="#lspci"><code>lspci</code></a> in a linux system.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ lspci | grep -i ethernet
02:00.<strong>0</strong> Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
02:00.<strong>1</strong> Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)</pre>
</div>
</div>
<div class="paragraph">
<p>The last number <strong>0</strong> or <strong>1</strong> here, is called a "function", which usually maps to
a physical port:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12697255/f9d1a552-c74c-11e5-93bd-77be10dc59b3.png" alt="f9d1a552 c74c 11e5 93bd 77be10dc59b3">
</div>
</div>
</div>
<div class="sect3">
<h4 id="_pci_functions_defined_in_sr_iov"><a class="anchor" href="#_pci_functions_defined_in_sr_iov"></a>17.1.3. PCI functions defined in SR-IOV</h4>
<div class="paragraph">
<p>SR-IOV introduces two new function types:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>Physical Functions (PFs)</strong> are full PCIe devices that include the SR-IOV
capabilities. <strong>Physical Functions</strong> are discovered, managed, and configured as
normal PCI devices. <strong>Physical Functions</strong> configure and manage the SR-IOV
functionality by assigning Virtual Functions.</p>
</li>
<li>
<p><strong>Virtual Functions (VFs)</strong> are simple PCIe functions that only process I/O.
These are "lightweight" PCIe functions that contain the resources necessary for
data movement but have a carefully minimized set of configuration resources.
Each <strong>Virtual Function</strong> is derived from a <strong>Physical Function</strong>. The number of
<strong>Virtual Functions</strong> a device may have is limited by the device hardware. A
single Ethernet port, the Physical Device, may map to many <strong>Virtual Functions</strong>
that can be shared to virtual machines.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The hypervisor can map one or more <strong>Virtual Functions</strong> to a virtual machine. The
<strong>Virtual Function</strong> 's configuration space is then mapped to the configuration
space presented to the guest.</p>
</div>
<div class="paragraph">
<p>in another word: with SR-IOV we can "split" one single physical NIC port into
multiple "virtual NIC port" , and each virtual NIC port can then be assigned to
different VMs.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="http://cdn.ttgtmedia.com/rms/onlineImages/SR-IOV.jpg" alt="SR IOV">
</div>
</div>
<div class="paragraph">
<p>Each Virtual Function can only be mapped to a single guest at a time, as
Virtual Functions require real hardware resources. A virtual machine can have
multiple Virtual Functions. A Virtual Function appears as a network card in the
same way as a normal network card would appear to an operating system.</p>
</div>
<div class="paragraph">
<p>The SR-IOV drivers are implemented in the kernel. The core implementation is
contained in the PCI subsystem, but there must also be driver support for both
the Physical Function (PF) and Virtual Function (VF) devices. An SR-IOV capable
device can allocate VFs from a PF. The VFs appear as PCI devices which are
backed on the physical PCI device by resources such as queues and register
sets. illustrated below:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12071301/afd9c7f4-b072-11e5-9916-302ff60b6148.png" alt="afd9c7f4 b072 11e5 9916 302ff60b6148">
</div>
<div class="title">Figure 8. A typical network adapter supporting SR-IOV functionality</div>
</div>
<div class="ulist">
<ul>
<li>
<p>pDRV: Physical Driver(<code>ixgbe</code> for Intel 82599 NIC used in our setup)</p>
</li>
<li>
<p>vDRV: Virtual Driver(<code>ixgbevf</code> for Intel 82599 NIC used in our setup)</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="_what_is_a_root"><a class="anchor" href="#_what_is_a_root"></a>17.1.4. what is a "root"?</h4>
<div class="paragraph">
<p>This is a PCIe term. A <code>root complex</code> connects the processor and memory
subsystem to the PCIe switch fabric composed of one or more switch devices,
similar to a host bridge in a PCI system, which generates transaction requests
on behalf of the processor interconnected through a local bus, and may contain
more than one PCIe port and multiple switch devices. A root Port is the portion
of the motherboard that contains the host bridge, which allows the PCIe ports
to talk to the rest of the computer.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12698268/8736f316-c766-11e5-94f7-7299db0ef9b9.png" alt="8736f316 c766 11e5 94f7 7299db0ef9b9">
</div>
<div class="title">Figure 9. root port</div>
</div>
</div>
<div class="sect3">
<h4 id="_why_is_it_called_single"><a class="anchor" href="#_why_is_it_called_single"></a>17.1.5. why is it called "single"?</h4>
<div class="paragraph">
<p>This is to differentiate it with another IO virtualization technology named
"MR-IOV" - Multiple Root IOV, which is defined to share IO resource on multiple
HW domains. For example, Multiple servers &amp; VMs sharing one I/O adapter, which
can be placed into a separate chassis outside of the server. Bandwidth of the
I/O adapter is shared among the servers.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12698734/79087a9c-c772-11e5-8f21-76dfab860cb3.png" alt="MR-IOV">
</div>
</div>
<div class="paragraph">
<p>a typical MR-IOV topology will look like:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12698755/4601358e-c773-11e5-8cb5-f4480ac1c5bf.png" alt="MR-IOV">
</div>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12698361/bfce4b00-c768-11e5-99f3-8441a322f75b.png" alt="MR-IOV">
</div>
</div>
<div class="paragraph">
<p>a brief comparison between SR-IOV and MR-IOV:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12698759/62eb77ae-c773-11e5-8391-88d0beedca00.png" alt="MR-IOV">
</div>
</div>
</div>
<div class="sect3">
<h4 id="_corelation_with_code_vt_d_code"><a class="anchor" href="#_corelation_with_code_vt_d_code"></a>17.1.6. corelation with <code>VT-d</code></h4>
<div class="paragraph">
<p>Intel’s implementation of PCI-SIG SR-IOV functionality requires the
VMM software to configure the direct assignment of the virtual
function to the virtual machine using Intel® Virtualization Technology
for Directed I/O (Intel® VT-d). Memory Translation technologies in
Intel® VT-d provide hardware assisted techniques to allow direct
DMA transfers. Intel VT-d helps with secure translation, and SRIOV
provides separate data spaces for the virtual machines. <sup class="footnote">[<a id="_footnoteref_6" class="footnote" href="#_footnote_6" title="View footnote.">6</a>]</sup></p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12072444/6574c7a6-b0ac-11e5-8a5f-3b6e861d8fa8.png" alt="6574c7a6 b0ac 11e5 8a5f 3b6e861d8fa8">
</div>
<div class="title">Figure 10. SR-IOV vs. VT-d</div>
</div>
<div class="paragraph">
<p>In brief, SR-IOV aims to "partition" or "split" the NIC port into VFs, while
<code>VT-d</code> aims to "assign" each VF to different VM. If the intention is to always
use the entire bandwidth and processing capability of a NIC, then VT-d can be
used without SR-IOV.  see section of <a href="#VT-D">VT-d</a> for more detailed
information about VT-d.</p>
</div>
</div>
<div class="sect3">
<h4 id="_other_requirement"><a class="anchor" href="#_other_requirement"></a>17.1.7. other requirement</h4>
<div class="paragraph">
<p>currently SR-IOV needs to be supported by different components in a server:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>BIOS support</p>
</li>
<li>
<p>HyperVisor support</p>
</li>
<li>
<p>NIC and driver support</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>And, Juniper&#8217;s modified NIC kernel driver requires linux kernel 3.13. This may
be changed later to be compatible with newer kernel releases.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_sr_iov_packet_flow"><a class="anchor" href="#_sr_iov_packet_flow"></a>17.2. SR-IOV packet flow</h3>
<div class="paragraph">
<p>The <a href="#SR-IOV-prime">[SR-IOV-prime]</a> illustrated packet processing flow in SR-IOV:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12698837/cc0e13de-c775-11e5-938d-1f6634ec06cf.png" alt="SR-IOV packet flow">
</div>
<div class="title">Figure 11. SR-IOV packet flow</div>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>The Ethernet packet arrives at the Intel® Ethernet NIC.</p>
</li>
<li>
<p>The packet is sent to the Layer 2 sorter/switch/classifier.</p>
<div class="ulist">
<ul>
<li>
<p>This Layer 2 sorter is configured by the Master Driver(MD).</p>
</li>
<li>
<p>When either the MD or the VF Driver configure a MAC address or VLAN, this
Layer 2 sorter is configured.</p>
</li>
</ul>
</div>
</li>
<li>
<p>After being sorted by the Layer 2 Switch, the packet is placed into a
receive queue dedicated to the target VF.</p>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
For SR-IOV, MAC address configuration in the VF is important. Missing
this step will make SR-IOV and VF fail to work, or lead to unexpected behavior.
</td>
</tr>
</table>
</div>
</li>
<li>
<p>The DMA operation is initiated. The target memory address for the DMA
operation is defined within the descriptors in the VF, which have been
configured by the VF driver within the VM.</p>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>In order for SR-IOV to work, the <code>ixgbevf</code> driver kernel module needs to be
loaded also in the guest VM.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>root@localhost:~# lspci | grep 82599
00:06.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:08.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

root@localhost:~# lspci -vks 00:06.0
00:06.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
        Subsystem: Hewlett-Packard Company Device 17d3
        Flags: bus master, fast devsel, latency 0
        Memory at fe000000 (64-bit, prefetchable) [size=16K]
        Memory at fe004000 (64-bit, prefetchable) [size=16K]
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
        Kernel driver in use: igb_uio
        Kernel modules: ixgbevf</pre>
</div>
</div>
</td>
</tr>
</table>
</div>
</li>
<li>
<p>The DMA Operation has reached the chipset. Intel® VT-d, which has been
configured by the VMM then remaps the target DMA address from a virtual host
address to a physical host address. The DMA operation is completed; the
Ethernet packet is now in the memory space of the VM.</p>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
As mentioned earlier, Intel&#8217;s SR-IOV implementation requires VT-d to be
enabled.
</td>
</tr>
</table>
</div>
</li>
<li>
<p>The Intel® Ethernet NIC fires an interrupt, indicating a packet has arrived.
This interrupt is handled by the VMM.</p>
</li>
<li>
<p>The VMM fires a virtual interrupt to the VM, so that it is informed that the
packet has arrived.</p>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
this is why sometime it is called "interupt-driven" work mode.
</td>
</tr>
</table>
</div>
</li>
</ol>
</div>
</div>
<div class="sect2">
<h3 id="_sr_iov_configuration_in_vmx"><a class="anchor" href="#_sr_iov_configuration_in_vmx"></a>17.3. SR-IOV configuration in VMX</h3>
<div class="paragraph">
<p>in <code>vmx.conf</code> file, the <code>device-type : sriov</code> option will instruct the
installation script to configure <code>Virtual Function</code> (VF) on NIC card.</p>
</div>
<div class="paragraph">
<p>the actual CLI commands executed by the script to configure <code>SR-IOV</code> is very
simple: just reload the ixgbe module with <code>max_vfs</code> parameter:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo rmmod ixgbe;
sudo modprobe ixgbe max_vfs=1,1,1,1,1,1,1,1;</pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
just change the <code>max_vfs</code> parameter (here is <strong>1</strong>) to the needed number of
VF in your setup.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>In practice, There is a small problem here:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>The first command <code>rmmod ixgbe</code> will remove the kernel driver module</p>
</li>
<li>
<p>as a result all interfaces driven by the ixgbe module will be removed from
the platform as well.</p>
</li>
<li>
<p>In the case that the mgmt port is also ixgbe-driven, This will disconnect the
current telnet/ssh session and we&#8217;ll need to login via console (e.g through
<code>ilo</code> ).</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To workaround this and also to reduce the time of interuption to our telnet/ssh
session, I prefer to use below concatenated commands to configure SR-IOV FVs
immediately right after ixgbe module removed, and add our mgmt interface back
to the external bridge, etc, all in one go.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo rmmod ixgbevf; sudo rmmod ixgbe; \
sudo modprobe ixgbe max_vfs=1,1,1,1,1,1,1,1; \
sudo modprobe tun ; sleep 5; \
sudo brctl addif br-ext em1</pre>
</div>
</div>
<div class="paragraph">
<p>Below <code>interface</code> configuration in <code>vmx.conf</code> will specify the mapping between
VF in the host and NIC in the guest VM, and properties of the VF.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>    - interface            : ge-0/0/0
      port-speed-mbps      : 10000
      nic                  : p3p1
      mtu                  : 2000
      virtual-function     : 0
      mac-address          : "02:04.17:01:02:01"
      description          : "ge-0/0/0 connects to eth6"

    - interface            : ge-0/0/1
      port-speed-mbps      : 10000
      nic                  : p2p1
      mtu                  : 2000
      virtual-function     : 0
      mac-address          : "02:04.17:01:02:02"
      description          : "ge-0/0/1 connects to eth7"</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_example_of_1_vf"><a class="anchor" href="#_example_of_1_vf"></a>17.4. example of 1 VF</h3>
<div class="listingblock">
<div class="title">to list all PF and VFs, use <code>lspci</code></div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ lspci | grep -i &quot;ethernet&quot;
<span class="line-numbers"> 2</span>02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 3</span>02:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 4</span>02:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 5</span>02:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 6</span>06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 7</span>06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 8</span>06:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 9</span>06:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">10</span>21:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">11</span>21:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">12</span>21:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">13</span>21:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">14</span>23:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">15</span>23:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">16</span>23:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">17</span>23:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)</code></pre>
</div>
</div>
<div class="paragraph">
<p>kernel logs will display how many VFs have been enabled for each and every port.</p>
</div>
<div class="listingblock">
<div class="title">kernel logs:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ dmesg | grep SR-IOV
<span class="line-numbers"> 2</span>[203216.832237] ixgbe 0000:02:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 3</span>[203217.067941] ixgbe 0000:02:00.1 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 4</span>[203217.303815] ixgbe 0000:06:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 5</span>[203217.539589] ixgbe 0000:06:00.1 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 6</span>[203217.783265] ixgbe 0000:21:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 7</span>[203218.022982] ixgbe 0000:21:00.1 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 8</span>[203218.254658] ixgbe 0000:23:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers"> 9</span>[203218.490378] ixgbe 0000:23:00.1 (unregistered net_device): SR-IOV enabled with 1 VFs</code></pre>
</div>
</div>
<div class="paragraph">
<p>One way to find out PCI bus address of an interface name, is to use <code>ethtool</code>
with <code>-i</code> option:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ ethtool -i p2p1
<span class="line-numbers"> 2</span>driver: ixgbe
<span class="line-numbers"> 3</span>version: 3.19.1
<span class="line-numbers"> 4</span>firmware-version: 0x8000076f, 1.475.0
<span class="line-numbers"> 5</span>bus-info: 0000:06:00.0  #&lt;------
<span class="line-numbers"> 6</span>supports-statistics: yes
<span class="line-numbers"> 7</span>supports-test: yes
<span class="line-numbers"> 8</span>supports-eeprom-access: yes
<span class="line-numbers"> 9</span>supports-register-dump: yes
<span class="line-numbers">10</span>supports-priv-flags: no
<span class="line-numbers">11</span>
<span class="line-numbers">12</span>ping@trinity:~$ ethtool -i p3p1
<span class="line-numbers">13</span>driver: ixgbe
<span class="line-numbers">14</span>version: 3.19.1
<span class="line-numbers">15</span>firmware-version: 0x8000076f, 1.475.0
<span class="line-numbers">16</span>bus-info: 0000:23:00.0  #&lt;------
<span class="line-numbers">17</span>supports-statistics: yes
<span class="line-numbers">18</span>supports-test: yes
<span class="line-numbers">19</span>supports-eeprom-access: yes
<span class="line-numbers">20</span>supports-register-dump: yes
<span class="line-numbers">21</span>supports-priv-flags: no
<span class="line-numbers">22</span>ping@trinity:~$</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>The other option is to use <code>lshw</code> command, which can be used to list
vendor/manufacturer info of all interfaces in the platform, an example is shown
below:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@ubuntu1:~$ sudo lshw -class network | grep -iE &quot;bus info|logical name&quot;
<span class="line-numbers"> 2</span>       bus info: pci@0000:07:00.0
<span class="line-numbers"> 3</span>       logical name: p2p1
<span class="line-numbers"> 4</span>       bus info: pci@0000:07:00.1
<span class="line-numbers"> 5</span>       logical name: p2p2
<span class="line-numbers"> 6</span>       bus info: pci@0000:07:10.0
<span class="line-numbers"> 7</span>       bus info: pci@0000:07:10.1
<span class="line-numbers"> 8</span>       bus info: pci@0000:07:10.2
<span class="line-numbers"> 9</span>       bus info: pci@0000:07:10.3
<span class="line-numbers">10</span>       bus info: pci@0000:07:10.4
<span class="line-numbers">11</span>       bus info: pci@0000:07:10.5
<span class="line-numbers">12</span>       bus info: pci@0000:07:10.6
<span class="line-numbers">13</span>       bus info: pci@0000:07:10.7
<span class="line-numbers">14</span>       bus info: pci@0000:03:00.0
<span class="line-numbers">15</span>       logical name: em1
<span class="line-numbers">16</span>       bus info: pci@0000:03:00.1
<span class="line-numbers">17</span>       logical name: em2
<span class="line-numbers">18</span>       bus info: pci@0000:03:00.2
<span class="line-numbers">19</span>       logical name: em3
<span class="line-numbers">20</span>       bus info: pci@0000:03:00.3
<span class="line-numbers">21</span>       logical name: em4
<span class="line-numbers">22</span>       bus info: pci@0000:0a:00.0
<span class="line-numbers">23</span>       logical name: p3p1
<span class="line-numbers">24</span>       bus info: pci@0000:0a:00.1
<span class="line-numbers">25</span>       logical name: p3p2
<span class="line-numbers">26</span>       bus info: pci@0000:24:00.0
<span class="line-numbers">27</span>       logical name: p5p1
<span class="line-numbers">28</span>       bus info: pci@0000:24:00.1
<span class="line-numbers">29</span>       logical name: p5p2
<span class="line-numbers">30</span>       bus info: pci@0000:24:10.0
<span class="line-numbers">31</span>       bus info: pci@0000:24:10.1
<span class="line-numbers">32</span>       bus info: pci@0000:24:10.2
<span class="line-numbers">33</span>       bus info: pci@0000:24:10.3
<span class="line-numbers">34</span>       bus info: pci@0000:24:10.4
<span class="line-numbers">35</span>       bus info: pci@0000:24:10.5
<span class="line-numbers">36</span>       bus info: pci@0000:24:10.6
<span class="line-numbers">37</span>       bus info: pci@0000:24:10.7
<span class="line-numbers">38</span>       logical name: br-int-vmx1-nic
<span class="line-numbers">39</span>       logical name: vfp_int-vmx1
<span class="line-numbers">40</span>       logical name: vfp_ext-vmx1
<span class="line-numbers">41</span>       logical name: br-ext-nic
<span class="line-numbers">42</span>       logical name: vcp_int-vmx1
<span class="line-numbers">43</span>       logical name: vcp_ext-vmx1</code></pre>
</div>
</div>
<div class="paragraph">
<p>the advantage of <code>lshw</code> over <code>ethtool</code> is, it list all network interfaces
hardware information instead of just one specific interfac, and, it also shows
the mapping relationship between PCI address and interface name, which comes
very handy.</p>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>with the bus address, the PF-VF mapping relationship can be printed:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:06\:00.0/virtfn*
<span class="line-numbers">2</span>lrwxrwxrwx 1 root root 0 Nov 23 21:03 /sys/bus/pci/devices/0000:06:00.0/virtfn0 -&gt; ../0000:06:10.0
<span class="line-numbers">3</span>
<span class="line-numbers">4</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:23\:00.0/virtfn*
<span class="line-numbers">5</span>lrwxrwxrwx 1 root root 0 Nov 23 21:03 /sys/bus/pci/devices/0000:23:00.0/virtfn0 -&gt; ../0000:23:10.0</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">to get all PF-VF map in a platform, "find" it like this:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo find /sys -name virtfn* | xargs ls -l
<span class="line-numbers"> 2</span>... /sys/.../0000:02:00.0/virtfn0 -&gt; ../0000:02:10.0
<span class="line-numbers"> 3</span>... /sys/.../0000:02:00.1/virtfn0 -&gt; ../0000:02:10.1
<span class="line-numbers"> 4</span>... /sys/.../0000:06:00.0/virtfn0 -&gt; ../0000:06:10.0
<span class="line-numbers"> 5</span>... /sys/.../0000:06:00.1/virtfn0 -&gt; ../0000:06:10.1
<span class="line-numbers"> 6</span>... /sys/.../0000:21:00.0/virtfn0 -&gt; ../0000:21:10.0
<span class="line-numbers"> 7</span>... /sys/.../0000:21:00.1/virtfn0 -&gt; ../0000:21:10.1
<span class="line-numbers"> 8</span>... /sys/.../0000:23:00.0/virtfn0 -&gt; ../0000:23:10.0
<span class="line-numbers"> 9</span>... /sys/.../0000:23:00.1/virtfn0 -&gt; ../0000:23:10.1</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>another method to find out VF-PF mapping corelation is to use libvirt <code>virsh</code>
command:</p>
</div>
<div class="paragraph">
<p>list running VMs and locate VCP name/ID:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ sudo virsh list
 Id    Name                           State
----------------------------------------------------
 34    vhepe-vcp                      running
 37    vhepe-vfp                      running</pre>
</div>
</div>
<div class="paragraph">
<p>list VFs(normally with slot number '0x10' or '0x11' ) in use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ sudo virsh dumpxml 37 | grep -iE "0x10|0x11"
        &lt;address type='pci' domain='0x0000' bus='0x24' slot='0x10' function='0x0'/&gt;
        &lt;address type='pci' domain='0x0000' bus='0x24' slot='0x10' function='0x1'/&gt;
        &lt;address type='pci' domain='0x0000' bus='0x07' slot='0x10' function='0x0'/&gt;
        &lt;address type='pci' domain='0x0000' bus='0x07' slot='0x10' function='0x1'/&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>locate the nodedev name and print the mapping relationship</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@ubuntu1:~$ sudo virsh nodedev-list | grep 0000_24_10
<span class="line-numbers"> 2</span>pci_0000_24_10_0
<span class="line-numbers"> 3</span>pci_0000_24_10_1
<span class="line-numbers"> 4</span>pci_0000_24_10_2
<span class="line-numbers"> 5</span>pci_0000_24_10_3
<span class="line-numbers"> 6</span>pci_0000_24_10_4
<span class="line-numbers"> 7</span>pci_0000_24_10_5
<span class="line-numbers"> 8</span>pci_0000_24_10_6
<span class="line-numbers"> 9</span>pci_0000_24_10_7
<span class="line-numbers">10</span>
<span class="line-numbers">11</span>ping@ubuntu1:~$ sudo virsh nodedev-dumpxml pci_0000_24_10_0 | grep function
<span class="line-numbers">12</span>    &lt;function&gt;0&lt;/function&gt;
<span class="line-numbers">13</span>    &lt;capability type='phys_function'&gt;
<span class="line-numbers">14</span>      &lt;address domain='0x0000' bus='0x24' slot='0x00' function='0x0'/&gt; <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">15</span>      &lt;address domain='0x0000' bus='0x24' slot='0x10' function='0x0'/&gt; <i class="conum" data-value="2"></i><b>(2)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>PF address</td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>VF address</td>
</tr>
</table>
</div>
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>if we put all info collected from above commands, we can produce a table that
lists the mapping relationship between PF, VF PCI bus address, logical
interface name, and the VMX (guest VM) virtual interface name, like this:</p>
</div>
<table class="tableblock frame-all grid-all spread">
<caption class="title">Table 5. NIC mapping table (1 VF)</caption>
<colgroup>
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">PCI address</th>
<th class="tableblock halign-left valign-top">adapter</th>
<th class="tableblock halign-left valign-top">interface</th>
<th class="tableblock halign-left valign-top">VF address</th>
<th class="tableblock halign-left valign-top">VF#</th>
<th class="tableblock halign-left valign-top">VMX interface</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em9</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p2p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p2p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">em2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p3p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:00.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p3p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>It will be very handy to have this table in hand, before you proceed to plan
your VMX installation. It will become even helpful if you plan to setup a
multi-instances VMX lab involving multiple VFs being configured and used. In
below sections I&#8217;ll demonstrate examples of configuring 4 VFs and 8 VFs.</p>
</div>
</div>
<div class="sect2">
<h3 id="_example_of_4_vfs"><a class="anchor" href="#_example_of_4_vfs"></a>17.5. example of 4 VFs</h3>
<div class="paragraph">
<p>To enable maximum of 4 VFs, give <code>max_vfs</code> number of <strong>4</strong> instead of <strong>1</strong> and
repeat the same exact commands:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers">1</span>sudo rmmod ixgbevf; sudo rmmod ixgbe; \
<span class="line-numbers">2</span>sudo modprobe ixgbe max_vfs=4,4,4,4,4,4,4,4; \
<span class="line-numbers">3</span>sudo modprobe tun ; sleep 5; \
<span class="line-numbers">4</span>sudo brctl addif br-ext em1</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">list all generated VFs:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ lspci | grep -i ethernet
<span class="line-numbers"> 2</span>02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 3</span>02:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 4</span>02:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 5</span>...&lt;snippet&gt;...
<span class="line-numbers"> 6</span>02:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 7</span>06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 8</span>06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 9</span>06:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">10</span>06:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">11</span>06:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">12</span>06:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">13</span>06:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">14</span>21:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">15</span>21:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">16</span>21:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">17</span>...&lt;snippet&gt;...
<span class="line-numbers">18</span>21:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">19</span>23:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">20</span>23:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">21</span>23:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">22</span>23:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">23</span>23:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">24</span>23:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">25</span>23:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">the kernel logs (dmesg) now show 4 VFs enabled</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>[26029.301160] ixgbe 0000:02:00.0 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">2</span>[26029.540597] ixgbe 0000:02:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">3</span>[26029.671906] ixgbe 0000:06:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers">4</span>[26029.908841] ixgbe 0000:06:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">5</span>[26030.144053] ixgbe 0000:21:00.0 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">6</span>[26030.383774] ixgbe 0000:21:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">7</span>[26030.515073] ixgbe 0000:23:00.0 (unregistered net_device): SR-IOV enabled with 1 VFs
<span class="line-numbers">8</span>[26030.752200] ixgbe 0000:23:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs</code></pre>
</div>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
Not all NIC ports now have 4 VFs , which will be explained later.
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="title">number of VFs currently enabled, vs number of maximum VFs supported, per NIC port:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo find /sys/ -name &quot;*vfs*&quot;
<span class="line-numbers"> 2</span>/sys/devices/pci0000:00/0000:00:02.2/0000:02:00.0/sriov_numvfs
<span class="line-numbers"> 3</span>/sys/devices/pci0000:00/0000:00:02.2/0000:02:00.0/sriov_totalvfs
<span class="line-numbers"> 4</span>/sys/devices/pci0000:00/0000:00:02.2/0000:02:00.1/sriov_numvfs
<span class="line-numbers"> 5</span>/sys/devices/pci0000:00/0000:00:02.2/0000:02:00.1/sriov_totalvfs
<span class="line-numbers"> 6</span>/sys/devices/pci0000:00/0000:00:03.0/0000:06:00.0/sriov_numvfs
<span class="line-numbers"> 7</span>/sys/devices/pci0000:00/0000:00:03.0/0000:06:00.0/sriov_totalvfs
<span class="line-numbers"> 8</span>/sys/devices/pci0000:00/0000:00:03.0/0000:06:00.1/sriov_numvfs
<span class="line-numbers"> 9</span>/sys/devices/pci0000:00/0000:00:03.0/0000:06:00.1/sriov_totalvfs
<span class="line-numbers">10</span>/sys/devices/pci0000:20/0000:20:02.2/0000:21:00.0/sriov_numvfs
<span class="line-numbers">11</span>/sys/devices/pci0000:20/0000:20:02.2/0000:21:00.0/sriov_totalvfs
<span class="line-numbers">12</span>/sys/devices/pci0000:20/0000:20:02.2/0000:21:00.1/sriov_numvfs
<span class="line-numbers">13</span>/sys/devices/pci0000:20/0000:20:02.2/0000:21:00.1/sriov_totalvfs
<span class="line-numbers">14</span>/sys/devices/pci0000:20/0000:20:03.0/0000:23:00.0/sriov_numvfs
<span class="line-numbers">15</span>/sys/devices/pci0000:20/0000:20:03.0/0000:23:00.0/sriov_totalvfs
<span class="line-numbers">16</span>/sys/devices/pci0000:20/0000:20:03.0/0000:23:00.1/sriov_numvfs
<span class="line-numbers">17</span>/sys/devices/pci0000:20/0000:20:03.0/0000:23:00.1/sriov_totalvfs
<span class="line-numbers">18</span>/sys/kernel/debug/tracing/events/vfs
<span class="line-numbers">19</span>
<span class="line-numbers">20</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:00/0000:00:02.2/0000:02:00.0/sriov_numvfs
<span class="line-numbers">21</span>4
<span class="line-numbers">22</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:00/0000:00:02.2/0000:02:00.0/sriov_totalvfs
<span class="line-numbers">23</span>63
<span class="line-numbers">24</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:00/0000:00:03.0/0000:06:00.0/sriov_numvfs
<span class="line-numbers">25</span>1
<span class="line-numbers">26</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:00/0000:00:03.0/0000:06:00.0/sriov_totalvfs
<span class="line-numbers">27</span>63
<span class="line-numbers">28</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:00/0000:00:03.0/0000:06:00.1/sriov_numvfs
<span class="line-numbers">29</span>4
<span class="line-numbers">30</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:00/0000:00:03.0/0000:06:00.1/sriov_totalvfs
<span class="line-numbers">31</span>63
<span class="line-numbers">32</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:02.2/0000:21:00.0/sriov_numvfs
<span class="line-numbers">33</span>4
<span class="line-numbers">34</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:02.2/0000:21:00.0/sriov_totalvfs
<span class="line-numbers">35</span>63
<span class="line-numbers">36</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:02.2/0000:21:00.1/sriov_numvfs
<span class="line-numbers">37</span>4
<span class="line-numbers">38</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:02.2/0000:21:00.1/sriov_totalvfs
<span class="line-numbers">39</span>63
<span class="line-numbers">40</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:03.0/0000:23:00.0/sriov_numvfs
<span class="line-numbers">41</span>1
<span class="line-numbers">42</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:03.0/0000:23:00.0/sriov_totalvfs
<span class="line-numbers">43</span>63
<span class="line-numbers">44</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:03.0/0000:23:00.1/sriov_numvfs
<span class="line-numbers">45</span>4
<span class="line-numbers">46</span>ping@trinity:~$ sudo cat /sys/devices/pci0000:20/0000:20:03.0/0000:23:00.1/sriov_totalvfs
<span class="line-numbers">47</span>63</code></pre>
</div>
</div>
<div class="paragraph">
<p>VF-PF mapping:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:02\:00.0/virtfn*
<span class="line-numbers"> 2</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.0/virtfn0 -&gt; ../0000:02:10.0
<span class="line-numbers"> 3</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.0/virtfn1 -&gt; ../0000:02:10.2
<span class="line-numbers"> 4</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.0/virtfn2 -&gt; ../0000:02:10.4
<span class="line-numbers"> 5</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.0/virtfn3 -&gt; ../0000:02:10.6
<span class="line-numbers"> 6</span>ping@trinity:~$
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:02\:00.1/virtfn*
<span class="line-numbers"> 9</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.1/virtfn0 -&gt; ../0000:02:10.1
<span class="line-numbers">10</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.1/virtfn1 -&gt; ../0000:02:10.3
<span class="line-numbers">11</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.1/virtfn2 -&gt; ../0000:02:10.5
<span class="line-numbers">12</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:02:00.1/virtfn3 -&gt; ../0000:02:10.7
<span class="line-numbers">13</span>ping@trinity:~$
<span class="line-numbers">14</span>
<span class="line-numbers">15</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:06\:00.0/rtfn*
<span class="line-numbers">16</span>lrwxrwxrwx 1 root root 0 Nov 21 11:57 /sys/bus/pci/devices/0000:06:00.0/virtfn0 -&gt; ../0000:06:10.0
<span class="line-numbers">17</span>
<span class="line-numbers">18</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:06:00.1/virtfn*
<span class="line-numbers">19</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:06:00.1/virtfn0 -&gt; ../0000:06:10.1
<span class="line-numbers">20</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:06:00.1/virtfn1 -&gt; ../0000:06:10.3
<span class="line-numbers">21</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:06:00.1/virtfn2 -&gt; ../0000:06:10.5
<span class="line-numbers">22</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:06:00.1/virtfn3 -&gt; ../0000:06:10.7
<span class="line-numbers">23</span>
<span class="line-numbers">24</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:21:00.0/virtfn*
<span class="line-numbers">25</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.0/virtfn0 -&gt; ../0000:21:10.0
<span class="line-numbers">26</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.0/virtfn1 -&gt; ../0000:21:10.2
<span class="line-numbers">27</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.0/virtfn2 -&gt; ../0000:21:10.4
<span class="line-numbers">28</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.0/virtfn3 -&gt; ../0000:21:10.6
<span class="line-numbers">29</span>
<span class="line-numbers">30</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:21:00.1/virtfn*
<span class="line-numbers">31</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.1/virtfn0 -&gt; ../0000:21:10.1
<span class="line-numbers">32</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.1/virtfn1 -&gt; ../0000:21:10.3
<span class="line-numbers">33</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.1/virtfn2 -&gt; ../0000:21:10.5
<span class="line-numbers">34</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:21:00.1/virtfn3 -&gt; ../0000:21:10.7
<span class="line-numbers">35</span>
<span class="line-numbers">36</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:23:00.0/virtfn*
<span class="line-numbers">37</span>lrwxrwxrwx 1 root root 0 Nov 21 11:57 /sys/bus/pci/devices/0000:23:00.0/virtfn0 -&gt; ../0000:23:10.0
<span class="line-numbers">38</span>
<span class="line-numbers">39</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:23:00.1/virtfn*
<span class="line-numbers">40</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:23:00.1/virtfn0 -&gt; ../0000:23:10.1
<span class="line-numbers">41</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:23:00.1/virtfn1 -&gt; ../0000:23:10.3
<span class="line-numbers">42</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:23:00.1/virtfn2 -&gt; ../0000:23:10.5
<span class="line-numbers">43</span>lrwxrwxrwx 1 root root 0 Nov 21 18:22 /sys/bus/pci/devices/0000:23:00.1/virtfn3 -&gt; ../0000:23:10.7</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>the PCI bus address may look different on different server, so correct PCI
address has to be used to execute each command above. A more convenient and
general way to display all PF-VF mapping is to <code>find</code> them out under <code>/sys</code>
folder.  below is a sample captured from a different server.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@ubuntu1:~$ sudo find /sys -name virtfn* | xargs ls -l
<span class="line-numbers"> 2</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.0/virtfn0 -&gt; ../0000:07:10.0
<span class="line-numbers"> 3</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.0/virtfn1 -&gt; ../0000:07:10.2
<span class="line-numbers"> 4</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.0/virtfn2 -&gt; ../0000:07:10.4
<span class="line-numbers"> 5</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.0/virtfn3 -&gt; ../0000:07:10.6
<span class="line-numbers"> 6</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.1/virtfn0 -&gt; ../0000:07:10.1
<span class="line-numbers"> 7</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.1/virtfn1 -&gt; ../0000:07:10.3
<span class="line-numbers"> 8</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.1/virtfn2 -&gt; ../0000:07:10.5
<span class="line-numbers"> 9</span>... /sys/devices/pci0000:00/0000:00:01.0/0000:07:00.1/virtfn3 -&gt; ../0000:07:10.7
<span class="line-numbers">10</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.0/virtfn0 -&gt; ../0000:24:10.0
<span class="line-numbers">11</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.0/virtfn1 -&gt; ../0000:24:10.2
<span class="line-numbers">12</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.0/virtfn2 -&gt; ../0000:24:10.4
<span class="line-numbers">13</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.0/virtfn3 -&gt; ../0000:24:10.6
<span class="line-numbers">14</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.1/virtfn0 -&gt; ../0000:24:10.1
<span class="line-numbers">15</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.1/virtfn1 -&gt; ../0000:24:10.3
<span class="line-numbers">16</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.1/virtfn2 -&gt; ../0000:24:10.5
<span class="line-numbers">17</span>... /sys/devices/pci0000:20/0000:20:02.2/0000:24:00.1/virtfn3 -&gt; ../0000:24:10.7</code></pre>
</div>
</div>
<div class="paragraph">
<p>Similiarly,
to print the number of currently enabled VFs on each interfaces:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:/export/home/vhepe/images$ sudo find /sys/ -name "*numvfs" | xargs cat
4
4
0
0
4
4</pre>
</div>
</div>
<div class="paragraph">
<p>to print the capability of support maximum VFs on each port:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@ubuntu1:/export/home/vhepe/images$ sudo find /sys/ -name &quot;*totalvfs&quot; | xargs cat
<span class="line-numbers">2</span>64
<span class="line-numbers">3</span>64
<span class="line-numbers">4</span>64
<span class="line-numbers">5</span>64
<span class="line-numbers">6</span>64
<span class="line-numbers">7</span>64</code></pre>
</div>
</div>
</td>
</tr>
</table>
</div>
<table class="tableblock frame-all grid-all spread">
<caption class="title">Table 6. NIC mapping table (4 VFs)</caption>
<colgroup>
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">PCI address</th>
<th class="tableblock halign-left valign-top">adapter</th>
<th class="tableblock halign-left valign-top">interface</th>
<th class="tableblock halign-left valign-top">VF address</th>
<th class="tableblock halign-left valign-top">VF#</th>
<th class="tableblock halign-left valign-top">VMX interface</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">02:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em9</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">02:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p2p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">06:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">p2p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">21:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">21:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:00.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">p3p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">23:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">p3p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>Now, the reason that <code>p2p1</code> and <code>p3p1</code> still hold just 1 VF instead of 4 in
this example, is because the guest VMX VMs were not torn down at the time when
SR-IOV VFs were reconfigured, so the previous VF resources were still hold
unchanged on the ports in use.</p>
</div>
<div class="paragraph">
<p>After removing the VMs and reconfiguring ixgbe the expected number of VFs can
be seen:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>sudo virsh destroy vcp-vmx1
<span class="line-numbers"> 2</span>sudo virsh destroy vfp-vmx1
<span class="line-numbers"> 3</span>sudo virsh undefine vcp-vmx1
<span class="line-numbers"> 4</span>sudo virsh undefine vfp-vmx1
<span class="line-numbers"> 5</span>sudo rmmod ixgbevf; sudo rmmod ixgbe; \
<span class="line-numbers"> 6</span>sudo modprobe ixgbe max_vfs=4,4,4,4,4,4,4,4; \
<span class="line-numbers"> 7</span>sudo modprobe tun ; sleep 5; sudo brctl addif em1
<span class="line-numbers"> 8</span>
<span class="line-numbers"> 9</span>[206499.970842] ixgbe 0000:02:00.0 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">10</span>[206500.206573] ixgbe 0000:02:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">11</span>[206500.446766] ixgbe 0000:06:00.0 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">12</span>[206500.682536] ixgbe 0000:06:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">13</span>[206500.917846] ixgbe 0000:21:00.0 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">14</span>[206501.153595] ixgbe 0000:21:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">15</span>[206501.389801] ixgbe 0000:23:00.0 (unregistered net_device): SR-IOV enabled with 4 VFs
<span class="line-numbers">16</span>[206501.625554] ixgbe 0000:23:00.1 (unregistered net_device): SR-IOV enabled with 4 VFs</code></pre>
</div>
</div>
<div class="sect3">
<h4 id="_the_rule_of_thumb_for_4_vf"><a class="anchor" href="#_the_rule_of_thumb_for_4_vf"></a>17.5.1. the "rule of thumb" for 4 VF</h4>
<div class="paragraph">
<p>The previous table for "4 VFs" scenario will now be updated as below:</p>
</div>
<table class="tableblock frame-all grid-all spread">
<caption class="title">Table 7. NIC mapping table</caption>
<colgroup>
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">PCI address</th>
<th class="tableblock halign-left valign-top">adapter</th>
<th class="tableblock halign-left valign-top">interface</th>
<th class="tableblock halign-left valign-top">VF address</th>
<th class="tableblock halign-left valign-top">VF#</th>
<th class="tableblock halign-left valign-top">VMX interface</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">02:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em9</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">02:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">06:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">p2p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">06:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">p2p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">21:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">21:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">em2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">23:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">p3p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">23:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="4"><p class="tableblock">p3p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>From this table, we seem to be able to conclude the following rules:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a number of <strong>10</strong> represents "virtual" <code>device</code> (or <code>bus</code>)</p>
</li>
<li>
<p>a sequence of "interleave" function numbers <strong>0 2 4 6</strong> represent the 4
"virtual" function <strong>0</strong> to <strong>3</strong> in the first physical port 0</p>
</li>
<li>
<p>an "interleave" numbers <strong>1 3 5 7</strong> represent the 4 "virtual" function <strong>0</strong> to
<strong>3</strong> in the second physical port 1</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>A question is : will this always hold true for other maximum VFs configuration?
A quick test can show the answer.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_example_of_8_vfs"><a class="anchor" href="#_example_of_8_vfs"></a>17.6. example of 8 VFs:</h3>
<div class="paragraph">
<p>As 1 VF and 4 VF examples shown above, to enable 8 VF per port (making it total
64 VFs platform wise), just change the <code>max_vfs</code> accordingly and use same
commands will be sufficient:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo rmmod ixgbevf; \
sudo rmmod ixgbe; \
sudo modprobe ixgbe max_vfs=8,8,8,8,8,8,8,8; \
sudo modprobe tun ; \
sleep 5; sudo brctl addif br-ext em1</pre>
</div>
</div>
<div class="paragraph">
<p>Now we can verify all the VF related data points using same commands as those
used in previous examples.</p>
</div>
<div class="listingblock">
<div class="title">to list all generated VFs</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/images/vmx_20151102.0/build$ lspci | grep -i ethernet
<span class="line-numbers"> 2</span>02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 3</span>02:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers"> 4</span>02:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 5</span>...&lt;snippet&gt;...
<span class="line-numbers"> 6</span>02:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 7</span>02:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers"> 8</span>...&lt;snippet&gt;...
<span class="line-numbers"> 9</span>02:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">10</span>06:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">11</span>06:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">12</span>06:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">13</span>...&lt;snippet&gt;...
<span class="line-numbers">14</span>06:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">15</span>06:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">16</span>...&lt;snippet&gt;...
<span class="line-numbers">17</span>06:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">18</span>21:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">19</span>21:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">20</span>21:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">21</span>...&lt;snippet&gt;...
<span class="line-numbers">22</span>21:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">23</span>21:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">24</span>...&lt;snippet&gt;...
<span class="line-numbers">25</span>21:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">26</span>23:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">27</span>23:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01)
<span class="line-numbers">28</span>23:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">29</span>...&lt;snippet&gt;...
<span class="line-numbers">30</span>23:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">31</span>23:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
<span class="line-numbers">32</span>...&lt;snippet&gt;...
<span class="line-numbers">33</span>23:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">kernel message:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:/images/vmx_20151102.0/build$ dmesg | grep SR-IOV
<span class="line-numbers"> 2</span>[72130.750581] ixgbe 0000:02:00.0 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 3</span>[72130.986275] ixgbe 0000:02:00.1 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 4</span>[72131.223191] ixgbe 0000:06:00.0 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 5</span>[72131.458462] ixgbe 0000:06:00.1 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 6</span>[72131.693388] ixgbe 0000:21:00.0 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 7</span>[72131.937491] ixgbe 0000:21:00.1 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 8</span>[72132.173784] ixgbe 0000:23:00.0 (unregistered net_device): SR-IOV enabled with 8 VFs
<span class="line-numbers"> 9</span>[72132.409462] ixgbe 0000:23:00.1 (unregistered net_device): SR-IOV enabled with 8 VFs</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">PF-VF mapping:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ ls -l /sys/bus/pci/devices/0000\:02\:00.0/virtfn*
<span class="line-numbers"> 2</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn0 -&gt; ../0000:02:10.0
<span class="line-numbers"> 3</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn1 -&gt; ../0000:02:10.2
<span class="line-numbers"> 4</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn2 -&gt; ../0000:02:10.4
<span class="line-numbers"> 5</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn3 -&gt; ../0000:02:10.6
<span class="line-numbers"> 6</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn4 -&gt; ../0000:02:11.0
<span class="line-numbers"> 7</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn5 -&gt; ../0000:02:11.2
<span class="line-numbers"> 8</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn6 -&gt; ../0000:02:11.4
<span class="line-numbers"> 9</span>lrwxrwxrwx 1 root root 0 Nov 24 14:10 /sys/bus/pci/devices/0000:02:00.0/virtfn7 -&gt; ../0000:02:11.6</code></pre>
</div>
</div>
<div class="sect3">
<h4 id="_the_rule_of_thumb_for_8_vf"><a class="anchor" href="#_the_rule_of_thumb_for_8_vf"></a>17.6.1. the "rule of thumb" for 8 VF</h4>
<div class="paragraph">
<p>Now we end up with below table for "8 VFs" scenario:</p>
</div>
<table class="tableblock frame-all grid-all spread">
<caption class="title">Table 8. NIC mapping table (8 VFs)</caption>
<colgroup>
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
<col style="width: 16%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">PCI address</th>
<th class="tableblock halign-left valign-top">adapter</th>
<th class="tableblock halign-left valign-top">interface</th>
<th class="tableblock halign-left valign-top">VF address</th>
<th class="tableblock halign-left valign-top">VF#</th>
<th class="tableblock halign-left valign-top">VMX interface</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">02:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">em9</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">02:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">em10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">02:11.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">06:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">p2p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">06:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">p2p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">06:11.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">21:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">em1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">21:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560FLB</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">em2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">21:11.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">23:00.0</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">p3p1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ge-0/0/0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">23:00.1</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">560M</p></td>
<td class="tableblock halign-left valign-top" rowspan="8"><p class="tableblock">p3p2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:10.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">23:11.7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>From this table, it looks the previous conclusions we got were almost right ,
except now we add one more device number <strong>11</strong>, so the rules is updated below:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a number of <strong>10</strong> and <strong>11</strong> represents "virtual" <code>device</code> (or <code>bus</code>)</p>
</li>
<li>
<p>a sequence of "interleave" function numbers <strong>0 2 4 6</strong> represent the 4
"virtual" function <strong>0</strong> to <strong>3</strong> under each virtual <code>device</code> number <strong>10</strong> and
<strong>11</strong>, in the first physical port 0</p>
</li>
<li>
<p>a sequence of "interleave" function numbers <strong>1 3 5 7</strong> represent the 4
"virtual" function <strong>0</strong> to <strong>3</strong> under each virtual virtual <code>device</code> number <strong>10</strong>
and <strong>11</strong> in the second physical port 1</p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="VIRTIO"><a class="anchor" href="#VIRTIO"></a>18. virtio</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_virtio_basic_concept"><a class="anchor" href="#_virtio_basic_concept"></a>18.1. virtio basic concept</h3>
<div class="paragraph">
<p>comparing with "full virtualization" that QEMU software provided, <code>virtio</code>
provides a "paravirtualization" environment, where guest operating system is
"aware" that its running on a Hypervisor and cooperates with the hypervisor.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12077041/8c519726-b199-11e5-9caa-97bdfcae6900.png" alt="8c519726 b199 11e5 9caa 97bdfcae6900">
</div>
<div class="title">Figure 12. Device emulation in full virtualization and paravirtualization environments</div>
</div>
<div class="paragraph">
<p><code>Virtio</code> is a virtualization standard for network and disk device drivers,
which enables guests VM to get high performance network and disk operations,
and gives most of the performance benefits of paravirtualization.</p>
</div>
<div class="paragraph">
<p>The key features of <code>virtio</code> are highlighted below:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Virtio is an abstraction of common emulated devices like PCI, Hard drive and
NICs</p>
</li>
<li>
<p>guest OS needs to have virtio drivers, called "front end", to be able to work
in this environment. examples of this front end drivers are:</p>
<div class="ulist">
<ul>
<li>
<p><code>virtio-blk</code></p>
</li>
<li>
<p><code>virtio-net</code></p>
</li>
<li>
<p><code>virtio-balloon</code>.</p>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
currently <code>ubuntu</code> and most other linux variations have <code>virtio</code> driver
supported in kernel hence no need any extra installations. for windows or other
OSs, virtio drivers may need to be installed seperately.
</td>
</tr>
</table>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>The hypervisor itself implements "backend drivers" for device emulation</p>
</li>
<li>
<p>These "front end" and "backend drivers" work together to make up <code>virtio</code></p>
</li>
<li>
<p>Usually front end drivers are part of QEMU and back end drivers are part of
KVM</p>
</li>
</ul>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12077037/44b0174e-b199-11e5-8774-4c544395d45e.png" alt="virtio">
</div>
<div class="title">Figure 13. driver abstraction with virtio</div>
</div>
<div class="paragraph">
<p>In addition to the front-end drivers (implemented in the guest operating
system) and the back-end drivers (implemented in the hypervisor), virtio
defines two layers to support guest-to-hypervisor communication:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>At the top level (called virtio) is the "virtual queue" interface that
conceptually attaches front-end drivers to back-end drivers.</p>
</li>
<li>
<p><code>virtio-ring</code> is used to buffer the info processed by the frontend/backend
driver. it can also buffer the I/O requests from the frontend driver before
hand over to the backend, improving the I/O processing efficiency.</p>
</li>
</ul>
</div>
<div class="imageblock">
<div class="content">
<img src="https://cloud.githubusercontent.com/assets/2038044/12077154/0e05dcee-b1a1-11e5-9042-138b40f4a01f.png" alt="0e05dcee b1a1 11e5 9042 138b40f4a01f">
</div>
<div class="title">Figure 14. High-level architecture of the virtio framework</div>
</div>
<div class="paragraph">
<p><code>virtio</code> can be used to create virtual NIC interfaces, independent of back end
drivers in HyperVisor.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
even For SR-IOV based MX86, virtio is still in use, but it is used only
for vRE/vPFE internal and external management interfaces, not for data plane
interfaces
</td>
</tr>
</table>
</div>
<div class="paragraph">
<div class="title">virtio support in kvm</div>
<p><code>Virtio</code> was chosen to be the main platform for IO virtualization in KVM. The
idea behind it is to have a common framework for hypervisors for IO
virtualization. At the moment, network/block/balloon devices are suported for
kvm. The host implementation is in userspace - qemu, so no driver is needed in
the host.</p>
</div>
<div class="paragraph">
<p>Virtio is relatively new technique and so it is not supported from first day of
qemu-kvm, according to <a href="#linux-kvm-virtio">[linux-kvm-virtio]</a>, kvm version needs to be <code>&gt;60</code> and
linux kernel needs to be later than <code>2.6.25</code>.  Also it needs some specific
configuration options to be activated in the kernel. To verify if the QEMU-KVM
installed in the server support virtio, run this command:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ qemu-system-x86_64 -net nic,model=?
qemu: Supported NIC models: ne2k_pci,i82551,i82557b,i82559er,rtl8139,e1000,pcnet,virtio
                                                                                 ^^^^^^ <i class="conum" data-value="1"></i><b>(1)</b></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>virtio is supported in this qemu-kvm release.</td>
</tr>
</table>
</div>
<div class="paragraph">
<div class="title">virtio in VMX</div>
<p>the installation of virtio version of VMX does not enforce the following
requirements:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>physical NIC with SR-IOV capability</p>
</li>
<li>
<p>IXGBE driver</p>
</li>
<li>
<p>specific linux kernel version (e.g. 3.13)</p>
</li>
<li>
<p>IOMMU/VT-d</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Therefore if performance is not a concern, it&#8217;s more convenient to setup
<code>virtio</code> version of VMX for learning/testing purpose.</p>
</div>
<div class="paragraph">
<div class="title">virtio linux kernel driver support</div>
<p>Current linux kernel already has <code>virtio</code> driver module support:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ grep -i virtio /boot/config-3.13.0-32-generic
<span class="line-numbers"> 2</span>CONFIG_NET_9P_VIRTIO=m
<span class="line-numbers"> 3</span>CONFIG_VIRTIO_BLK=y
<span class="line-numbers"> 4</span>CONFIG_SCSI_VIRTIO=m
<span class="line-numbers"> 5</span>CONFIG_VIRTIO_NET=y
<span class="line-numbers"> 6</span>CONFIG_CAIF_VIRTIO=m
<span class="line-numbers"> 7</span>CONFIG_VIRTIO_CONSOLE=y
<span class="line-numbers"> 8</span>CONFIG_HW_RANDOM_VIRTIO=m
<span class="line-numbers"> 9</span>CONFIG_VIRTIO=y
<span class="line-numbers">10</span># Virtio drivers
<span class="line-numbers">11</span>CONFIG_VIRTIO_PCI=y
<span class="line-numbers">12</span>CONFIG_VIRTIO_BALLOON=y
<span class="line-numbers">13</span>CONFIG_VIRTIO_MMIO=y
<span class="line-numbers">14</span>CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
<span class="line-numbers">15</span>
<span class="line-numbers">16</span>ping@trinity:~$ find /lib/modules/3.13.0-32-generic/ -name &quot;virtio*&quot;
<span class="line-numbers">17</span>/lib/modules/3.13.0-32-generic/kernel/drivers/scsi/virtio_scsi.ko
<span class="line-numbers">18</span>/lib/modules/3.13.0-32-generic/kernel/drivers/char/hw_random/virtio-rng.ko</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_virtio_verification_in_vpfe_guest_vm"><a class="anchor" href="#_virtio_verification_in_vpfe_guest_vm"></a>18.2. virtio verification in vPFE guest VM</h3>
<div class="paragraph">
<p>logging into vPFE VM and more details about virtio can be observed.</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>loggin into vPFE</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ telnet localhost 8817
<span class="line-numbers">2</span>Trying ::1...
<span class="line-numbers">3</span>Trying 127.0.0.1...
<span class="line-numbers">4</span>Connected to localhost.
<span class="line-numbers">5</span>Escape character is '^]'.
<span class="line-numbers">6</span>Wind River Linux 6.0.0.12 vfp-vmx1 console
<span class="line-numbers">7</span>vfp-vmx1 login: pfe
<span class="line-numbers">8</span>Password:</code></pre>
</div>
</div>
</li>
<li>
<p>para-virtualization</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p><code>para-virtualization</code> simply means guest VM is "aware of" the virtualization
environment, which can be verified by looking at the vFPC guest VM booting log:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>/boot/modules/virtio.ko size 0x69a0 at 0x1616000
<span class="line-numbers"> 2</span>/boot/modules/virtio_pci.ko size 0x6fb8 at 0x161d000
<span class="line-numbers"> 3</span>/boot/modules/virtio_blk.ko size 0x7988 at 0x1624000
<span class="line-numbers"> 4</span>/boot/modules/if_vtnet.ko size 0x34f10 at 0x162c000
<span class="line-numbers"> 5</span>/boot/modules/virtio_console.ko size 0x9740 at 0x1661000
<span class="line-numbers"> 6</span>
<span class="line-numbers"> 7</span>virtio_pci0: &lt;VirtIO PCI Network adapter&gt; port 0xc560-0xc57f mem
<span class="line-numbers"> 8</span>    0xfebf1000-0xfebf1fff irq 10 at device 5.0 on pci0
<span class="line-numbers"> 9</span>em1: &lt;VirtIO Networking Adapter&gt; on virtio_pci0
<span class="line-numbers">10</span>virtio_pci0: host features: 0x511fffe3
<span class="line-numbers">11</span>    &lt;RingIndirect,NotifyOnEmpty,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,
<span class="line-numbers">12</span>    MrgRxBuf,TxUFO,TxTSOECN,TxTSOv6,TxTSOv4,RxUFO,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,
<span class="line-numbers">13</span>    MacAddress,RxChecksum,TxChecksum&gt;
<span class="line-numbers">14</span>virtio_pci0: negotiated features: 0x110f8020 &lt;RingIndirect,NotifyOnEmpty,
<span class="line-numbers">15</span>    VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,MacAddress&gt;
<span class="line-numbers">16</span>virtio_pci1: &lt;VirtIO PCI Balloon adapter&gt; port 0xc580-0xc59f irq 10 at device
<span class="line-numbers">17</span>    6.0 on pci0</code></pre>
</div>
</div>
</div>
</div>
</li>
<li>
<p>virtio NIC</p>
<div class="paragraph">
<p>To view the emulated PCI devices, use the same <code>lspci</code> command in the vPFE
guest VM, as what we did in the host:</p>
</div>
<div class="openblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre>pfe@vfp-vmx1:~$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device         <i class="conum" data-value="1"></i><b>(1)</b>
00:04.0 Ethernet controller: Red Hat, Inc Virtio network device         <i class="conum" data-value="2"></i><b>(2)</b>
00:05.0 Ethernet controller: Red Hat, Inc Virtio network device         <i class="conum" data-value="3"></i><b>(3)</b>
00:06.0 Ethernet controller: Red Hat, Inc Virtio network device         <i class="conum" data-value="4"></i><b>(4)</b>
00:07.0 Multimedia audio controller: Intel Corporation 82801AA AC'97 Audio Controller (rev 01)
00:08.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon</pre>
</div>
</div>
<div class="paragraph">
<p>The output captured here matches to the <code>virsh</code> command that we demonstrated
previously, listing all VM virtual network(VN) info using KVM qmp command <code>info
network</code> from outside of the VM (from host):</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh qemu-monitor-command vcp-vmx1 --hmp "info network"
[sudo] password for ping:
net0: index=0,type=nic,model=e1000,macaddr=02:04:17:01:01:01
 \ hostnet0: index=0,type=tap,fd=18
net1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:10:91:fe
 \ hostnet1: index=0,type=tap,fd=19

ping@trinity:~$ sudo virsh qemu-monitor-command vfp-vmx1 --hmp "info network"
net0: index=0,type=nic,model=virtio-net-pci,macaddr=02:04:17:01:01:02   <i class="conum" data-value="1"></i><b>(1)</b>
 \ hostnet0: index=0,type=tap,fd=18
net1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:db:34:a9   <i class="conum" data-value="2"></i><b>(2)</b>
 \ hostnet1: index=0,type=tap,fd=21
net2: index=0,type=nic,model=virtio-net-pci,macaddr=02:04:17:01:02:01   <i class="conum" data-value="3"></i><b>(3)</b>
 \ hostnet2: index=0,type=tap,fd=23
net3: index=0,type=nic,model=virtio-net-pci,macaddr=02:04:17:01:02:02   <i class="conum" data-value="4"></i><b>(4)</b>
 \ hostnet3: index=0,type=tap,fd=25</pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>virtio interface for vPFE external mgmt interface: <code>ext</code></td>
</tr>
<tr>
<td><i class="conum" data-value="2"></i><b>2</b></td>
<td>virtio interface for vPFE internal mgmt interface: <code>int</code></td>
</tr>
<tr>
<td><i class="conum" data-value="3"></i><b>3</b></td>
<td>virtio interface for JUNOS interface <code>ge-0/0/0</code></td>
</tr>
<tr>
<td><i class="conum" data-value="4"></i><b>4</b></td>
<td>virtio interface for JUNOS interface <code>ge-0/0/1</code></td>
</tr>
</table>
</div>
<div class="paragraph">
<p>to get more detail about the virtio emulated PCI device:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>pfe@vfp-vmx1:~$ lspci -vvks 00:03.0
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
        Subsystem: Red Hat, Inc Device 0001
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast &gt;TAbort- &lt;TAbort- &lt;MAbort- &gt;SERR- &lt;PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 10
        Region 0: I/O ports at c520 [size=32]
        Region 1: Memory at febd1000 (32-bit, non-prefetchable) [size=4K]
        Expansion ROM at feac0000 [disabled] [size=256K]
        Capabilities: &lt;access denied&gt;
        Kernel driver in use: virtio-pci        #&lt;------</pre>
</div>
</div>
<div class="paragraph">
<p>as expected, currently <code>virtio</code> driver module is in use on this virtio emulated
virtual NICs.</p>
</div>
</div>
</div>
</li>
<li>
<p>virtio-balloon</p>
<div class="openblock">
<div class="content">
<div class="paragraph">
<p>besides NIC, virtio also brings a memory-saving technique called "balloon",
which will be covered in more details later.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>pfe@vfp-vmx1:~$ lspci -vvks 00:08.0
00:08.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
        Subsystem: Red Hat, Inc Device 0005
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast &gt;TAbort- &lt;TAbort- &lt;MAbort- &gt;SERR- &lt;PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 11
        Region 0: I/O ports at c5a0 [size=32]
        Kernel driver in use: virtio-pci</pre>
</div>
</div>
<div class="paragraph">
<p>TODO</p>
</div>
</div>
</div>
</li>
</ol>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_cpu_pinning_affinitization"><a class="anchor" href="#_cpu_pinning_affinitization"></a>19. cpu pinning (affinitization)</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_2_terms"><a class="anchor" href="#_2_terms"></a>19.1. 2 terms</h3>
<div class="paragraph">
<p>There are 2 related terms regarding CPU pinning:</p>
</div>
<div class="dlist">
<dl>
<dt class="hdlist1">CPU pinning</dt>
<dd>
<p>CPU pinning is the ability to run specific VM&#8217;s virtual CPU (vCPU) on specific
physical CPU (pCPU) in a specific host.</p>
</dd>
</dl>
</div>
<div class="paragraph">
<p>As explained in <a href="#VCPU_ESSENTIAL">vcpu essential</a>, a "vcpu" essentially is nothing but a
thread running inside the space of a QEMU process. Therefore, pinning a "vCPU"
is pinning a thread - restricting a thread to run on a dedicated physical CPU.</p>
</div>
<div class="paragraph">
<p>Restricting a thread to run on a single CPU avoids the performance cost caused
by the cache invalidation that occurs when a thread ceases to execute on one
CPU and then recommences execution on a different CPU. this is why CPU pinning
usually gain performance - if tuned properly.</p>
</div>
<div class="dlist">
<dl>
<dt class="hdlist1">CPU affinity</dt>
<dd>
<p>CPU affinity is a scheduler property that "bonds" a process to a given set of
CPUs on the system.  The Linux scheduler will honor the given CPU affinity and
the process will not run on any other CPUs. The  CPU affinity is represented as
a bitmask, with the lowest order bit corresponding to the first logical CPU and
the highest order bit corresponding to the last logical CPU.  The masks are
typically given in hexadecimal.</p>
</dd>
</dl>
</div>
<div class="paragraph">
<p>A thread&#8217;s <code>CPU affinity mask</code> determines the set of CPUs on which it is
eligible to run.  On a multiprocessor system, vCPU pinning can be internally
archived by setting the <code>CPU affinity mask</code>.</p>
</div>
<div class="paragraph">
<p>It is possible to ensure maximum execution speed for that thread,
by dedicating one CPU to a particular thread:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>setting the affinity mask of that thread to specify a single CPU</p>
</li>
<li>
<p>setting the affinity mask of all other threads to exclude that CPU)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>for example:</p>
</div>
<table class="tableblock frame-all grid-all" style="width: 50%;">
<colgroup>
<col style="width: 50%;">
<col style="width: 50%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">affinity mask</th>
<th class="tableblock halign-left valign-top">eligibile processors</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0x00000001</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">processor #0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0x00000003</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">processors #0 and #1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0xFFFFFFFF</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">all processors (#0 through #31).</p></td>
</tr>
</tbody>
</table>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>the Linux scheduler also supports natural CPU affinity: the scheduler  attempts
to  keep  pro‐ cesses on the same CPU as long as practical for performance
reasons.  Therefore, forcing a specific CPU affinity is useful only in certain
applications.</p>
</div>
<div class="paragraph">
<p>Not all CPUs may exist on a given system but a mask may specify more CPUs than
are present.  A retrieved mask will reflect only the bits that correspond to
CPUs physically on the system.</p>
</div>
<div class="paragraph">
<p>If an invalid mask is given (i.e., one that corresponds to no valid  CPUs  on
the current system) an error is returned.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_2_code_virsh_code_commands"><a class="anchor" href="#_2_code_virsh_code_commands"></a>19.2. 2 <code>virsh</code> commands</h3>
<div class="paragraph">
<p>There are at least 2 commands in <code>libvirt</code> <code>virsh</code> tool regarding "CPU
affinity" feature:</p>
</div>
<div class="dlist">
<dl>
<dt class="hdlist1"><code>vcpuinfo</code></dt>
<dd>
<p>used to retrieve the CPU affinity of a running VM.</p>
</dd>
<dt class="hdlist1"><code>vcpupin</code></dt>
<dd>
<p>used to "pin" guest domain virtual CPUs to physical host CPUs.</p>
</dd>
</dl>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
these 2 <code>virsh</code> commands play a very similiar roll as what linux utility
<code>taskset</code> does, which is used to set or retrieve the CPU affinity of a running
process given its PID or to launch a new COMMAND with a given CPU affinity.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_a_simple_test"><a class="anchor" href="#_a_simple_test"></a>19.3. a simple test</h3>
<div class="paragraph">
<p>here is a simple test to demonstrate this feature:</p>
</div>
<div class="paragraph">
<p>first, list vcpuinfo BEFORE vcpupin:</p>
</div>
<div class="listingblock">
<div class="title">vmx1 vRE:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">1</span>ping@trinity:~$ sudo virsh vcpuinfo vcp-vmx1
<span class="line-numbers">2</span>VCPU:           0
<span class="line-numbers">3</span>CPU:            0
<span class="line-numbers">4</span>State:          running
<span class="line-numbers">5</span>CPU time:       93.2s
<span class="line-numbers">6</span>CPU Affinity:   y----------------------------- <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">vmx1 vFPC:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh vcpuinfo vfp-vmx1
<span class="line-numbers"> 2</span>VCPU:           0
<span class="line-numbers"> 3</span>CPU:            6
<span class="line-numbers"> 4</span>State:          running
<span class="line-numbers"> 5</span>CPU time:       18.1s
<span class="line-numbers"> 6</span>CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>VCPU:           1
<span class="line-numbers"> 9</span>CPU:            24
<span class="line-numbers">10</span>State:          running
<span class="line-numbers">11</span>CPU time:       12.9s
<span class="line-numbers">12</span>CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">13</span>
<span class="line-numbers">14</span>VCPU:           2
<span class="line-numbers">15</span>CPU:            28
<span class="line-numbers">16</span>State:          running
<span class="line-numbers">17</span>CPU time:       12.4s
<span class="line-numbers">18</span>CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy <i class="conum" data-value="1"></i><b>(1)</b>
<span class="line-numbers">19</span>
<span class="line-numbers">20</span>VCPU:           3
<span class="line-numbers">21</span>CPU:            5
<span class="line-numbers">22</span>State:          running
<span class="line-numbers">23</span>CPU time:       12.2s
<span class="line-numbers">24</span>CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy <i class="conum" data-value="1"></i><b>(1)</b></code></pre>
</div>
</div>
<div class="colist arabic">
<table>
<tr>
<td><i class="conum" data-value="1"></i><b>1</b></td>
<td>"CPU Affinity mask". a <code>y</code> on a bit indicate the eligibility of the
specific CPU NO. to run current process.</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>as can be seen before CPU pinning, the 4 VCPU/threads can be running in any of
the physical cores, because the "CPU Affinity" is all <code>y</code>.</p>
</div>
<div class="paragraph">
<p>This is because by default, <code>libvirt</code> provisions guests using the hypervisor&#8217;s
default policy. For most hypervisors, the policy is to run guests on any
available processing core or CPU.</p>
</div>
<div class="paragraph">
<p>There are times when an explicit policy may be better, in particular for
systems with a NUMA (Non-Uniform Memory Access) architecture. A guest on a NUMA
system should be pinned to a processing core so that its memory allocations are
always local to the node it is running on. This avoids cross-node memory
transports which have less bandwidth and can significantly degrade performance.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>virsh vcpupin vfp-vmx1 0 11
virsh vcpupin vfp-vmx1 1 12
virsh vcpupin vfp-vmx1 2 13
virsh vcpupin vfp-vmx1 3 8
virsh vcpupin vfp-vmx1 4 9
virsh vcpupin vcp-vmx1 0 15
virsh emulatorpin vcp-vmx1 0
virsh emulatorpin vfp-vmx1 0</pre>
</div>
</div>
<div class="paragraph">
<p>This is the current mapping between vCPU(thread) and CPU (to shortly the
capture we ignored the <code>cpuinfo</code> output from vmx2 instance):</p>
</div>
<table class="tableblock frame-all grid-all" style="width: 50%;">
<colgroup>
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">vCPU(vmx1)</th>
<th class="tableblock halign-left valign-top">CPU</th>
<th class="tableblock halign-left valign-top">vCPU(vmx2)</th>
<th class="tableblock halign-left valign-top">CPU</th>
<th class="tableblock halign-left valign-top">Affinity</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0(vcp)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0(vcp)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">any CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">27</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">any CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">24</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">31</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">any CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">28</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">25</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">any CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">30</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">any CPU</p></td>
</tr>
</tbody>
</table>
<div class="listingblock">
<div class="title">after vcpupin</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers"> 1</span>ping@trinity:~$ sudo virsh vcpuinfo vcp-vmx1
<span class="line-numbers"> 2</span>VCPU:           0
<span class="line-numbers"> 3</span>CPU:            7
<span class="line-numbers"> 4</span>State:          running
<span class="line-numbers"> 5</span>CPU time:       465.7s
<span class="line-numbers"> 6</span>CPU Affinity:   -------y------------------------
<span class="line-numbers"> 7</span>
<span class="line-numbers"> 8</span>ping@trinity:~$ sudo virsh vcpuinfo vfp-vmx1
<span class="line-numbers"> 9</span>VCPU:           0
<span class="line-numbers">10</span>CPU:            0
<span class="line-numbers">11</span>State:          running
<span class="line-numbers">12</span>CPU time:       571.7s
<span class="line-numbers">13</span>CPU Affinity:   y-------------------------------
<span class="line-numbers">14</span>
<span class="line-numbers">15</span>VCPU:           1
<span class="line-numbers">16</span>CPU:            1
<span class="line-numbers">17</span>State:          running
<span class="line-numbers">18</span>CPU time:       1432.5s
<span class="line-numbers">19</span>CPU Affinity:   -y------------------------------
<span class="line-numbers">20</span>
<span class="line-numbers">21</span>VCPU:           2
<span class="line-numbers">22</span>CPU:            2
<span class="line-numbers">23</span>State:          running
<span class="line-numbers">24</span>CPU time:       1133.6s
<span class="line-numbers">25</span>CPU Affinity:   --y-----------------------------
<span class="line-numbers">26</span>
<span class="line-numbers">27</span>VCPU:           3
<span class="line-numbers">28</span>CPU:            3
<span class="line-numbers">29</span>State:          running
<span class="line-numbers">30</span>CPU time:       517.1s
<span class="line-numbers">31</span>CPU Affinity:   ---y----------------------------</code></pre>
</div>
</div>
<table class="tableblock frame-all grid-all" style="width: 50%;">
<colgroup>
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">vCPU(vmx1)</th>
<th class="tableblock halign-left valign-top">CPU</th>
<th class="tableblock halign-left valign-top">vCPU(vmx2)</th>
<th class="tableblock halign-left valign-top">CPU</th>
<th class="tableblock halign-left valign-top">Affinity</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0(vcp)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0(vcp)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">15</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">only specified CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">8</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">only specified CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">9</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">only specified CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">only specified CPU</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">11</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">only specified CPU</p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_hugepage"><a class="anchor" href="#_hugepage"></a>20. hugepage</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_hugepage_basic_concept"><a class="anchor" href="#_hugepage_basic_concept"></a>20.1. hugepage basic concept</h3>
<div class="paragraph">
<p>Whenever a process uses some memory, CPU will mark the RAM as used by that
process. For efficiency, x86 CPU allocate RAM by chunks of 4K bytes, which is
named as "a page". Those pages can be swapped to disk, etc.</p>
</div>
<div class="paragraph">
<p>Since the process address space are virtual, the CPU and the operating system
have to remember which page belong to which process, and where it is stored.
the data structure that is used to maintain this info is called <code>TLB</code>
(translation lookaside buffer) , which is often implemented as <code>CAM</code>
(content-addressable memory).</p>
</div>
<div class="paragraph">
<p>The CAM search key is the virtual address and the search result is a physical
address. If the requested address is present in the TLB, the CAM search yields
a match quickly and the retrieved physical address can be used to access
memory. This is called a <code>TLB hit</code>. If the requested address is not in the TLB,
it is a <code>cache miss</code>.</p>
</div>
<div class="paragraph">
<p>Obviously, the more pages you have, the more time it takes the CPU to search
the TLB for the target memory. Increasing page size (hence called <code>Hugepage</code>)
will reduce the amount of pages, and eventually improving the CPU performance.</p>
</div>
<div class="paragraph">
<p>Most current CPU architectures support hugepage, but possibly with a different
name/term, but they are all the same thing:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Huge pages (on Linux)</p>
</li>
<li>
<p>Super Pages (on BSD)</p>
</li>
<li>
<p>Large Pages (on Windows)</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="_hugepage_allocation"><a class="anchor" href="#_hugepage_allocation"></a>20.2. hugepage allocation</h3>
<div class="paragraph">
<p>The allocation of hugepages should be done at boot time or as soon as possible
after system boot to prevent memory from being fragmented in physical memory.
To reserve <code>hugepages</code> at boot time, a parameter is passed to the Linux* kernel
on the kernel command line.</p>
</div>
<div class="paragraph">
<p>x86 by default use 4K page size</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ getconf PAGESIZE
4096</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>root@ubuntu1:~# getconf -a | grep -i page
PAGESIZE                           4096
PAGE_SIZE                          4096
_AVPHYS_PAGES                      72309823
_PHYS_PAGES                        99076139</pre>
</div>
</div>
<div class="paragraph">
<p><code>VMX</code> use hugepage size <strong>2M</strong>, and total number of hugepages is configured to be
<code>32k</code>, make it 64G hugepage memory for the guest VMs.</p>
</div>
<div class="paragraph">
<p>before hugepage was enabled:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@trinity:/images/vmx_20151102.0/build$ cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB</pre>
</div>
</div>
<div class="paragraph">
<p>after:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>root@ubuntu1:~# grep -i page /proc/meminfo
AnonPages:       2244644 kB
PageTables:        17280 kB
AnonHugePages:   2076672 kB
HugePages_Total:   32786            #&lt;------
HugePages_Free:    24969
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB</pre>
</div>
</div>
<div class="paragraph">
<p>same info can be printed from kernel options using <code>sysctl</code> utility:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:/sys/module/ixgbevf$ sudo sysctl -a | grep -i hugepage
vm.hugepages_treat_as_movable = 0
vm.nr_hugepages = 32786     #&lt;------
vm.nr_hugepages_mempolicy = 32786
vm.nr_overcommit_hugepages = 0</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ sysctl vm.nr_hugepages
vm.nr_hugepages = 32786</pre>
</div>
</div>
<div class="paragraph">
<p>In ubuntu system, and all the other linux variants, these parameters are
stored in the below file systems:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>under <code>/proc/sys/vm/</code> folder in the <code>/proc</code> file system</p>
</li>
<li>
<p>under <code>/sys/kernel/mm/hugepages/</code> folder in the <code>/sys</code> file system</p>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ grep -R "" /proc/sys/vm/*huge*
/proc/sys/vm/hugepages_treat_as_movable:0
/proc/sys/vm/hugetlb_shm_group:0
/proc/sys/vm/nr_hugepages:32768
/proc/sys/vm/nr_hugepages_mempolicy:32768
/proc/sys/vm/nr_overcommit_hugepages:0</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ grep -R "" /sys/kernel/mm/hugepages/
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages:0
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages:32768
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages_mempolicy:32768
/sys/kernel/mm/hugepages/hugepages-2048kB/surplus_hugepages:0
/sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages:0
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages:24951
ping@ubuntu1:~$</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="sidebarblock">
<div class="content">
<div class="title">about <code>/proc</code> and <code>/sys</code></div>
<div class="paragraph">
<p><code>/proc</code> is very special in that it is also a virtual filesystem. It&#8217;s sometimes
referred to as a process information pseudo-file system. It doesn&#8217;t contain
'real' files but runtime system information (e.g. system memory, devices
mounted, hardware configuration, etc). For this reason it can be regarded as a
control and information centre for the kernel. In fact, quite a lot of system
utilities are simply calls to files in this directory. For example, 'lsmod' is
the same as 'cat /proc/modules' while 'lspci' is a synonym for 'cat /proc/pci'.
By altering files located in this directory you can even read/change kernel
parameters (sysctl) while the system is running. refer to
<a href="#proc_file_system">[proc_file_system]</a> for more details of proc file system.</p>
</div>
<div class="paragraph">
<p><code>sysfs</code> is a feature of the Linux 2.6 kernel that allows kernel code to export
information to user processes via an in-memory filesystem. The organization of
the filesystem directory hierarchy is strict, and based the internal
organization of kernel data structures. The files that are created in the
filesystem are (mostly) ASCII files with (usually) one value per file. These
features ensure that the information exported is accurate and easily
accessible, making sysfs one of the most intuitive and useful features of the
2.6 kernel.</p>
</div>
</div>
</div>
<div class="paragraph">
<p>Another tool to query the same data is <code>hugeadm</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ hugeadm --pool-list
      Size  Minimum  Current  Maximum  Default
   2097152    32768    32768    32768        *</pre>
</div>
</div>
<div class="paragraph">
<p>Interestingly, with <code>--explain</code> option it shows some "explanation" about the
output.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ hugeadm --pool-list --explain
      Size  Minimum  Current  Maximum  Default
   2097152    32768    32768    32768        *
Total System Memory: 387016 MB</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Mount Point          Options
/run/hugepages/kvm   rw,relatime,mode=775,gid=112
/HugePage_vPFE       rw,relatime
/HugePage_vPFE       rw,relatime</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Huge page pools:
      Size  Minimum  Current  Maximum  Default
   2097152    32768    32768    32768        *</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Huge page sizes with configured pools:</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>A /proc/sys/kernel/shmmax value of 33554432 bytes may be sub-optimal. To maximise
shared memory usage, this should be set to the size of the largest shared memory
segment size you want to be able to use. Alternatively, set it to a size matching
the maximum possible allocation size of all huge pages. This can be done
automatically, using the --set-recommended-shmmax option.</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>The recommended shmmax for your currently allocated huge pages is 68719476736 bytes.
To make shmmax settings persistent, add the following line to /etc/sysctl.conf:
  kernel.shmmax = 68719476736</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>hugeadm:WARNING: User ping (uid: 1003) is not a member of the hugetlb_shm_group root (gid: 0)!</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Note: Permanent swap space should be preferred when dynamic huge page pools are used.</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_change_hugepages_number"><a class="anchor" href="#_change_hugepages_number"></a>20.3. change hugepages number</h3>
<div class="paragraph">
<p>To change the number of hugepages in realtime, use kernel tool <code>sysctl</code>. in
this example we change hugepages from <code>32786</code> to <code>32768</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@ubuntu1:~$ sudo sysctl vm.nr_hugepages=32768
vm.nr_hugepages = 32768</pre>
</div>
</div>
<div class="paragraph">
<p>Now the number of total hugepages are changed, as well as the free hugepages:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ grep -i huge /proc/meminfo
AnonHugePages:   2076672 kB
HugePages_Total:   32768        #&lt;------
HugePages_Free:    24951        #&lt;------
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_hugepage_and_numa"><a class="anchor" href="#_hugepage_and_numa"></a>20.4. hugepage and numa</h3>
<div class="paragraph">
<p>The huage pages are distributed between all numanode, in this server there are
2 numanode , so each of node were allocated 16384 pages. the interesting part
is the number of free hugepages in each node: the hugepages from first node <code>0</code>
will be consumed first (by VMX instance in our context), so only <code>8567</code> pages
are left. VMX consumed <code>7817</code> pages, which is <code>15634M</code> memory.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ cat /sys/devices/system/node/node*/meminfo | grep Huge
Node 0 AnonHugePages:     88064 kB
Node 0 HugePages_Total: 16384   #&lt;------
Node 0 HugePages_Free:   8567
Node 0 HugePages_Surp:      0
Node 1 AnonHugePages:   2050048 kB
Node 1 HugePages_Total: 16384   #&lt;------
Node 1 HugePages_Free:  16384
Node 1 HugePages_Surp:      0</pre>
</div>
</div>
<div class="paragraph">
<p>The hugepages from the second node (node1) will be consumed if we start another
VMX instance. The hugepage allocation were performed by <code>libvirt</code> "under the
scene".</p>
</div>
</div>
<div class="sect2">
<h3 id="_a_simple_test_of_hugepage"><a class="anchor" href="#_a_simple_test_of_hugepage"></a>20.5. a simple test of hugepage</h3>
<div class="paragraph">
<p>currently <code>24951</code> hugepages are free:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ grep -i huge /proc/meminfo
AnonHugePages:   2076672 kB
HugePages_Total:   32768
HugePages_Free:    24951        #&lt;------
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB</pre>
</div>
</div>
<div class="paragraph">
<p>if we destroy VPFE VM:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ sudo virsh destroy 36
Domain 36 destroyed</pre>
</div>
</div>
<div class="paragraph">
<p>now all hugepages were claimed back:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ grep -i huge /proc/meminfo
AnonHugePages:   2068480 kB
HugePages_Total:   32768
HugePages_Free:    32768        #&lt;------
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB</pre>
</div>
</div>
<div class="paragraph">
<p>start VM:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ sudo virsh start vhepe-vfp
Domain vhepe-vfp started</pre>
</div>
</div>
<div class="paragraph">
<p>free huagepages were again consumed by the VM.</p>
</div>
<div class="listingblock">
<div class="content">
<pre>ping@ubuntu1:~$ grep -i huge /proc/meminfo
AnonHugePages:   2076672 kB
HugePages_Total:   32768
HugePages_Free:    24951        #&lt;------
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB</pre>
</div>
</div>
<div class="paragraph">
<p>totally <code>(32768-24951) X 2048K = 15634M</code> hugepage memory were allocated to the
vPFE VM.</p>
</div>
<div class="paragraph">
<p>refer to <a href="https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt">huge page</a>
for more details on this topic.</p>
</div>
<div style="page-break-after: always;"></div>
<div style="page-break-after: always;"></div>
</div>
</div>
</div>
<h1 id="appendix" class="sect0"><a class="anchor" href="#appendix"></a>appendix</h1>
<div class="sect1">
<h2 id="_reference"><a class="anchor" href="#_reference"></a>reference</h2>
<div class="sectionbody">
<div class="ulist">
<div class="title">VMX official document</div>
<ul>
<li>
<p><a href="https://www.juniper.net/support/downloads/?p=vmx">VMX official document/software download</a></p>
</li>
<li>
<p><a href="http://www.juniper.net/techpubs/en_US/vmx15.1/information-products/topic-collections/release-notes/index.html">vmx 15.1F3 release note</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">intel spec</div>
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/X86_virtualization">x86 virtualization</a></p>
</li>
<li>
<p><a href="http://www8.hp.com/us/en/products/iss-adapters/product-detail.html?oid=5274097">HP Ethernet 10Gb 2-port 560FLB Adapter</a></p>
</li>
<li>
<p><a href="http://www.intel.com/content/www/us/en/embedded/products/networking/82599-10-gbe-controller-datasheet.html">Intel 82599 10GE controller</a></p>
</li>
<li>
<p><a href="http://ark.intel.com/products/75287/Intel-Xeon-Processor-E5-4627-v2-16M-Cache-3_30-GHz">E5-4627 spec</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">libvirt/virsh</div>
<ul>
<li>
<p><a href="http://wiki.libvirt.org/page/FAQ#Libvirt_FAQ">libvirt FAQ</a></p>
</li>
<li>
<p><a href="https://libvirt.org/formatnetwork.html#examplesRoute">libvirt network XML format</a></p>
</li>
<li>
<p><a href="http://libvirt.org/sources/virshcmdref/html-single/">Virsh Command Reference</a></p>
</li>
<li>
<p><a href="https://www.redhat.com/archives/libvir-list/2014-September/msg00055.html">libvirt 1.2.8 release</a></p>
</li>
<li>
<p><a href="http://wiki.libvirt.org/page/VirtualNetworking">libvirt virtual networking</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">virtio</div>
<ul>
<li>
<p><a href="http://wiki.libvirt.org/page/Virtio">libvirt virtio</a></p>
</li>
<li>
<p><a href="http://www.ibm.com/developerworks/library/l-virtio/">Virtio: An I/O virtualization framework for Linux</a></p>
</li>
<li>
<p><a id="linux-kvm-virtio"></a>[linux-kvm-virtio] <a href="http://www.linux-kvm.org/page/Virtio">virtio</a></p>
</li>
<li>
<p><a href="http://www.cse.iitd.ernet.in/~sbansal/csl862-virt/2010/readings/p95-russell.pdf">virtio: Towards a De-Facto Standard For Virtual I/O Devices</a></p>
</li>
<li>
<p><a href="https://rwmj.wordpress.com/2010/07/17/virtio-balloon/">virtio balloon</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">VT-d/PCI passthrough</div>
<ul>
<li>
<p><a href="http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM">How to assign devices with VT-d in KVM</a></p>
</li>
<li>
<p><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/chap-Virtualization-PCI_passthrough.html">PCI passthrough with virsh</a></p>
</li>
<li>
<p><a href="http://www.ibm.com/developerworks/linux/library/l-pci-passthrough/">linux PCI passthough</a></p>
</li>
<li>
<p><a href="http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf">VT-d spec</a></p>
</li>
<li>
<p><a href="https://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices">Intel VT-d</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">SR-IOV</div>
<ul>
<li>
<p><a href="http://www8.hp.com/h20195/v2/GetPDF.aspx/4AA5-7050ENW.pdf">Implementing SR-IOV for Linux on HP ProLiant servers</a></p>
</li>
<li>
<p><a id="SR-IOV-prime"></a> <a href="http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf">PCI-SIG SR-IOV Primer</a></p>
</li>
<li>
<p><a href="http://www.intel.com/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf">SR-IOV white paper</a></p>
</li>
<li>
<p><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/chap-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV.html">SR-IOV</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">KVM/qemu features</div>
<ul>
<li>
<p><a href="https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt">huge page</a></p>
</li>
<li>
<p><a href="https://launchpad.net/ubuntu/trusty/+source/qemu/+changelog">qemu changelog</a></p>
</li>
<li>
<p><a href="http://wiki.qemu.org/QMP#QEMU_Machine_Protocol">qmp</a></p>
</li>
<li>
<p><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-NUMA-NUMA_and_libvirt.html">LIBVIRT NUMA TUNING</a></p>
</li>
<li>
<p><a href="https://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/ch25s06.html">Setting KVM processor affinities</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">DPDK</div>
<ul>
<li>
<p><a href="http://www.intel.com/content/dam/www/public/us/en/documents/guides/dpdk-getting-started-guide.pdf">dpdk-getting-started-guide</a></p>
</li>
<li>
<p><a href="http://dpdk.org/doc/nics">dpdk supported nics</a></p>
</li>
</ul>
</div>
<div class="ulist">
<div class="title">misc</div>
<ul>
<li>
<p><a id="BDF"></a> <a href="http://wiki.xenproject.org/wiki/Bus:Device.Function_(BDF)_Notation">BDF notation</a></p>
</li>
<li>
<p><a id="proc_file_system"></a> <a href="http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html">proc file system</a></p>
</li>
<li>
<p><a id="sys_file_system"></a>[sys_file_system] <a href="https://www.kernel.org/pub/linux/kernel/people/mochel/doc/papers/ols-2005/mochel.pdf">sys file system</a></p>
</li>
<li>
<p><a href="https://computing.llnl.gov/tutorials/pthreads/#Thread">posix thread programming</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="folder-structure"><a class="anchor" href="#folder-structure"></a>VMX package folder structure</h2>
<div class="sectionbody">
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="linux"><span class="line-numbers">  1</span>ping@trinity:~/vmx_20151102.0$ tree
<span class="line-numbers">  2</span>.
<span class="line-numbers">  3</span>├── config                          #&lt;------all config files that will be read
<span class="line-numbers">  4</span>│   ├── samples                             and parsed by the installation script
<span class="line-numbers">  5</span>│   │   ├── vmx.conf.sriov
<span class="line-numbers">  6</span>│   │   ├── vmx.conf.virtio
<span class="line-numbers">  7</span>│   │   └── vmx-galaxy.conf
<span class="line-numbers">  8</span>│   ├── vmx.conf                    #&lt;------the main config file used
<span class="line-numbers">  9</span>│   └── vmx-junosdev.conf
<span class="line-numbers"> 10</span>├── docs
<span class="line-numbers"> 11</span>│   ├── mx86_on_openstack_release_notes.pdf
<span class="line-numbers"> 12</span>│   └── VMX_Release_Notes_Installation_Guide_Beta.pdf
<span class="line-numbers"> 13</span>├── drivers
<span class="line-numbers"> 14</span>│   ├── galaxy
<span class="line-numbers"> 15</span>│   │   ├── Makefile
<span class="line-numbers"> 16</span>│   │   └── network_add.c
<span class="line-numbers"> 17</span>│   └── ixgbe-3.19.1                #&lt;------ixgbe driver source code
<span class="line-numbers"> 18</span>│       ├── COPYING
<span class="line-numbers"> 19</span>│       ├── ixgbe.7
<span class="line-numbers"> 20</span>│       ├── ixgbe.spec
<span class="line-numbers"> 21</span>│       ├── pci.updates
<span class="line-numbers"> 22</span>│       ├── README
<span class="line-numbers"> 23</span>│       ├── scripts
<span class="line-numbers"> 24</span>│       │   └── set_irq_affinity
<span class="line-numbers"> 25</span>│       ├── src
<span class="line-numbers"> 26</span>│       │   ├── ixgbe.7.gz
<span class="line-numbers"> 27</span>│       │   ├── ixgbe_82598.c
<span class="line-numbers"> 28</span>│       │   ├── ixgbe_82598.h
<span class="line-numbers"> 29</span>│       │   ├── ixgbe_82599.c
<span class="line-numbers"> 30</span>│       │   ├── ixgbe_82599.h
<span class="line-numbers"> 31</span>│       │   ├── ixgbe_api.c
<span class="line-numbers"> 32</span>│       │   ├── ixgbe_api.h
<span class="line-numbers"> 33</span>│       │   ├── ixgbe_common.c
<span class="line-numbers"> 34</span>│       │   ├── ixgbe_common.h
<span class="line-numbers"> 35</span>│       │   ├── ixgbe_dcb_82598.c
<span class="line-numbers"> 36</span>│       │   ├── ixgbe_dcb_82598.h
<span class="line-numbers"> 37</span>│       │   ├── ixgbe_dcb_82599.c
<span class="line-numbers"> 38</span>│       │   ├── ixgbe_dcb_82599.h
<span class="line-numbers"> 39</span>│       │   ├── ixgbe_dcb.c
<span class="line-numbers"> 40</span>│       │   ├── ixgbe_dcb.h
<span class="line-numbers"> 41</span>│       │   ├── ixgbe_dcb_nl.c
<span class="line-numbers"> 42</span>│       │   ├── ixgbe_debugfs.c
<span class="line-numbers"> 43</span>│       │   ├── ixgbe_ethtool.c
<span class="line-numbers"> 44</span>│       │   ├── ixgbe_fcoe.c
<span class="line-numbers"> 45</span>│       │   ├── ixgbe_fcoe.h
<span class="line-numbers"> 46</span>│       │   ├── ixgbe.h
<span class="line-numbers"> 47</span>│       │   ├── ixgbe_lib.c
<span class="line-numbers"> 48</span>│       │   ├── ixgbe_main.c
<span class="line-numbers"> 49</span>│       │   ├── ixgbe_mbx.c
<span class="line-numbers"> 50</span>│       │   ├── ixgbe_mbx.h
<span class="line-numbers"> 51</span>│       │   ├── ixgbe.mod.c
<span class="line-numbers"> 52</span>│       │   ├── ixgbe_osdep2.h
<span class="line-numbers"> 53</span>│       │   ├── ixgbe_osdep.h
<span class="line-numbers"> 54</span>│       │   ├── ixgbe_param.c
<span class="line-numbers"> 55</span>│       │   ├── ixgbe_phy.c
<span class="line-numbers"> 56</span>│       │   ├── ixgbe_phy.h
<span class="line-numbers"> 57</span>│       │   ├── ixgbe_procfs.c
<span class="line-numbers"> 58</span>│       │   ├── ixgbe_ptp.c
<span class="line-numbers"> 59</span>│       │   ├── ixgbe_sriov.c
<span class="line-numbers"> 60</span>│       │   ├── ixgbe_sriov.h
<span class="line-numbers"> 61</span>│       │   ├── ixgbe_sysfs.c
<span class="line-numbers"> 62</span>│       │   ├── ixgbe_type.h
<span class="line-numbers"> 63</span>│       │   ├── ixgbe_x540.c
<span class="line-numbers"> 64</span>│       │   ├── ixgbe_x540.h
<span class="line-numbers"> 65</span>│       │   ├── kcompat.c
<span class="line-numbers"> 66</span>│       │   ├── kcompat_ethtool.c
<span class="line-numbers"> 67</span>│       │   ├── kcompat.h
<span class="line-numbers"> 68</span>│       │   ├── Makefile
<span class="line-numbers"> 69</span>│       │   ├── modules.order
<span class="line-numbers"> 70</span>│       │   ├── Module.supported
<span class="line-numbers"> 71</span>│       │   └── Module.symvers
<span class="line-numbers"> 72</span>│       └── SUMS
<span class="line-numbers"> 73</span>├── env
<span class="line-numbers"> 74</span>│   ├── ubuntu_sriov.env
<span class="line-numbers"> 75</span>│   └── ubuntu_virtio.env
<span class="line-numbers"> 76</span>├── images                          #&lt;------all VM images that will be
<span class="line-numbers"> 77</span>│   ├── metadata_usb.img                    loaded by the KVM-qumu hypervisor
<span class="line-numbers"> 78</span>│   ├── vFPC-20151102.img
<span class="line-numbers"> 79</span>│   └── vmxhdd.img
<span class="line-numbers"> 80</span>├── scripts                         #&lt;------installation scripts
<span class="line-numbers"> 81</span>│   ├── common
<span class="line-numbers"> 82</span>│   │   ├── get_numa.sh
<span class="line-numbers"> 83</span>│   │   ├── get_pci.sh
<span class="line-numbers"> 84</span>│   │   ├── vmx_common_utils.sh
<span class="line-numbers"> 85</span>│   │   ├── vmx_configure.py        #&lt;------python script to parse YAML config file
<span class="line-numbers"> 86</span>│   │   ├── vmx_env.sh                      and generate the libvirt friendly XML files
<span class="line-numbers"> 87</span>│   │   ├── vmx_galaxy.py
<span class="line-numbers"> 88</span>│   │   ├── vmx_img_cli.sh
<span class="line-numbers"> 89</span>│   │   └── vmx_preinstall_checks.sh
<span class="line-numbers"> 90</span>│   ├── junosdev-bind
<span class="line-numbers"> 91</span>│   │   ├── vmx_brctl.sh
<span class="line-numbers"> 92</span>│   │   ├── vmx-junosdev-bind.py
<span class="line-numbers"> 93</span>│   │   ├── vmx_linkctl.sh
<span class="line-numbers"> 94</span>│   │   └── vmx_vhost_pin.sh
<span class="line-numbers"> 95</span>│   ├── kvm
<span class="line-numbers"> 96</span>│   │   ├── common
<span class="line-numbers"> 97</span>│   │   │   ├── vmx_kvm_bringup.sh
<span class="line-numbers"> 98</span>│   │   │   ├── vmx_kvm_cleanup.sh
<span class="line-numbers"> 99</span>│   │   │   ├── vmx_kvm_system_setup.sh
<span class="line-numbers">100</span>│   │   │   └── vmx_kvm_verify.sh
<span class="line-numbers">101</span>│   │   ├── sriov
<span class="line-numbers">102</span>│   │   │   └── vmx_kvm_sriov.sh
<span class="line-numbers">103</span>│   │   └── virtio
<span class="line-numbers">104</span>│   │       └── vmx_kvm_virtio.sh
<span class="line-numbers">105</span>│   └── templates
<span class="line-numbers">106</span>│       ├── _br_ext-ref.xml
<span class="line-numbers">107</span>│       ├── _br_int-ref.xml
<span class="line-numbers">108</span>│       ├── _vPFE-ref-ubuntu.xml
<span class="line-numbers">109</span>│       ├── _vPFE-ref.xml
<span class="line-numbers">110</span>│       └── _vRE-ref.xml
<span class="line-numbers">111</span>└── vmx.sh                          #&lt;------the main script to be executed
<span class="line-numbers">112</span>
<span class="line-numbers">113</span>18 directories, 91 files</code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_vmx_installation_script_generated_xml_file"><a class="anchor" href="#_vmx_installation_script_generated_xml_file"></a>VMX installation script generated XML file</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="xml-files"><a class="anchor" href="#xml-files"></a>generated vRE xml</h3>
<div class="paragraph">
<p><code>build/vmx1/xml/vRE-generated.xml</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers">  1</span><span class="error">ping@trinity:~$ cat xml.backup/vRE-generated.xml</span>
<span class="line-numbers">  2</span><span class="error">&lt;domain type=&quot;kvm&quot;&gt;</span>
<span class="line-numbers">  3</span>
<span class="line-numbers">  4</span>  <span class="error">&lt;name&gt;vcp-vmx1&lt;/name&gt;                             #&lt;------domain name (VM)</span>
<span class="line-numbers">  5</span>
<span class="line-numbers">  6</span>  <span class="error">&lt;memory unit=&quot;Mb&quot;&gt;2048&lt;/memory&gt;                   #&lt;------memory: 2G</span>
<span class="line-numbers">  7</span>
<span class="line-numbers">  8</span>  <span class="error">&lt;vcpu placement=&quot;static&quot;&gt;1&lt;/vcpu&gt;                 #&lt;------1 CPU for RE</span>
<span class="line-numbers">  9</span>
<span class="line-numbers"> 10</span>  <span class="error">&lt;cputune&gt;                                         #&lt;------finetune: vcpupin</span>
<span class="line-numbers"> 11</span>    <span class="error">&lt;vcpupin cpuset=&quot;0&quot; vcpu=&quot;0&quot; /&gt;</span>
<span class="line-numbers"> 12</span>  <span class="error">&lt;/cputune&gt;</span>
<span class="line-numbers"> 13</span>
<span class="line-numbers"> 14</span>  <span class="error">&lt;resource&gt;</span>
<span class="line-numbers"> 15</span>    <span class="error">&lt;partition&gt;/machine&lt;/partition&gt;</span>
<span class="line-numbers"> 16</span>  <span class="error">&lt;/resource&gt;</span>
<span class="line-numbers"> 17</span>
<span class="line-numbers"> 18</span>  <span class="error">&lt;sysinfo type=&quot;smbios&quot;&gt;</span>
<span class="line-numbers"> 19</span>    <span class="error">&lt;bios&gt;</span>
<span class="line-numbers"> 20</span>      <span class="error">&lt;entry name=&quot;vendor&quot;&gt;Juniper&lt;/entry&gt;</span>
<span class="line-numbers"> 21</span>    <span class="error">&lt;/bios&gt;</span>
<span class="line-numbers"> 22</span>    <span class="error">&lt;system&gt;</span>
<span class="line-numbers"> 23</span>      <span class="error">&lt;entry name=&quot;manufacturer&quot;&gt;VMX&lt;/entry&gt;</span>
<span class="line-numbers"> 24</span>      <span class="error">&lt;entry name=&quot;product&quot;&gt;VM-vcp_vmx1-161-re-0&lt;/entry&gt;</span>
<span class="line-numbers"> 25</span>      <span class="error">&lt;entry name=&quot;version&quot;&gt;0.1.0&lt;/entry&gt;</span>
<span class="line-numbers"> 26</span>    <span class="error">&lt;/system&gt;</span>
<span class="line-numbers"> 27</span>  <span class="error">&lt;/sysinfo&gt;</span>
<span class="line-numbers"> 28</span>
<span class="line-numbers"> 29</span>  <span class="error">&lt;os&gt;                                              #&lt;------os info</span>
<span class="line-numbers"> 30</span>    <span class="error">&lt;smbios mode=&quot;sysinfo&quot; /&gt;</span>
<span class="line-numbers"> 31</span>    <span class="error">&lt;type arch=&quot;x86_64&quot; machine=&quot;pc-0.13&quot;&gt;hvm&lt;/type&gt;</span>
<span class="line-numbers"> 32</span>    <span class="error">&lt;boot dev=&quot;hd&quot; /&gt;                               #&lt;------boot sequence</span>
<span class="line-numbers"> 33</span>  <span class="error">&lt;/os&gt;</span>
<span class="line-numbers"> 34</span>
<span class="line-numbers"> 35</span>  <span class="error">&lt;features&gt;                                        #&lt;------HW featured supported</span>
<span class="line-numbers"> 36</span>    <span class="error">&lt;acpi /&gt;</span>
<span class="line-numbers"> 37</span>    <span class="error">&lt;apic /&gt;</span>
<span class="line-numbers"> 38</span>    <span class="error">&lt;pae /&gt;</span>
<span class="line-numbers"> 39</span>  <span class="error">&lt;/features&gt;</span>
<span class="line-numbers"> 40</span>
<span class="line-numbers"> 41</span>  <span class="error">&lt;cpu mode=&quot;host-model&quot;&gt;</span>
<span class="line-numbers"> 42</span>    <span class="error">&lt;topology cores=&quot;1&quot; sockets=&quot;1&quot; threads=&quot;1&quot; /&gt;</span>
<span class="line-numbers"> 43</span>  <span class="error">&lt;/cpu&gt;</span>
<span class="line-numbers"> 44</span>
<span class="line-numbers"> 45</span>  <span class="error">&lt;clock offset=&quot;utc&quot; /&gt;</span>
<span class="line-numbers"> 46</span>
<span class="line-numbers"> 47</span>  <span class="error">&lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;</span>
<span class="line-numbers"> 48</span>
<span class="line-numbers"> 49</span>  <span class="error">&lt;on_reboot&gt;restart&lt;/on_reboot&gt;</span>
<span class="line-numbers"> 50</span>
<span class="line-numbers"> 51</span>  <span class="error">&lt;on_crash&gt;restart&lt;/on_crash&gt;</span>
<span class="line-numbers"> 52</span>
<span class="line-numbers"> 53</span>  <span class="error">&lt;devices&gt;</span>
<span class="line-numbers"> 54</span>    <span class="error">&lt;emulator&gt;/usr/bin/qemu-system-x86_64&lt;/emulator&gt;</span>
<span class="line-numbers"> 55</span>
<span class="line-numbers"> 56</span>    <span class="error">&lt;disk device=&quot;disk&quot; type=&quot;file&quot;&gt;</span>
<span class="line-numbers"> 57</span>      <span class="error">&lt;driver cache=&quot;directsync&quot; name=&quot;qemu&quot; type=&quot;qcow2&quot; /&gt;</span>
<span class="line-numbers"> 58</span>      <span class="error">&lt;source file=&quot;/images/vmx_20151102.0/build/vmx1/images/jinstall64-vmx-15.1F-20151104.0-domestic.img&quot; /&gt;</span>
<span class="line-numbers"> 59</span>      <span class="error">&lt;target bus=&quot;ide&quot; dev=&quot;hda&quot; /&gt;</span>
<span class="line-numbers"> 60</span>      <span class="error">&lt;address bus=&quot;0&quot; controller=&quot;0&quot; target=&quot;0&quot; type=&quot;drive&quot; unit=&quot;0&quot; /&gt;</span>
<span class="line-numbers"> 61</span>    <span class="error">&lt;/disk&gt;</span>
<span class="line-numbers"> 62</span>
<span class="line-numbers"> 63</span>    <span class="error">&lt;disk device=&quot;disk&quot; type=&quot;file&quot;&gt;</span>
<span class="line-numbers"> 64</span>      <span class="error">&lt;driver cache=&quot;directsync&quot; name=&quot;qemu&quot; type=&quot;qcow2&quot; /&gt;</span>
<span class="line-numbers"> 65</span>      <span class="error">&lt;source file=&quot;/images/vmx_20151102.0/build/vmx1/images/vmxhdd.img&quot; /&gt;</span>
<span class="line-numbers"> 66</span>      <span class="error">&lt;target bus=&quot;ide&quot; dev=&quot;hdb&quot; /&gt;</span>
<span class="line-numbers"> 67</span>      <span class="error">&lt;address bus=&quot;0&quot; controller=&quot;0&quot; target=&quot;0&quot; type=&quot;drive&quot; unit=&quot;1&quot; /&gt;</span>
<span class="line-numbers"> 68</span>    <span class="error">&lt;/disk&gt;</span>
<span class="line-numbers"> 69</span>
<span class="line-numbers"> 70</span>    <span class="error">&lt;disk device=&quot;disk&quot; type=&quot;file&quot;&gt;</span>
<span class="line-numbers"> 71</span>      <span class="error">&lt;driver cache=&quot;directsync&quot; name=&quot;qemu&quot; type=&quot;raw&quot; /&gt;</span>
<span class="line-numbers"> 72</span>      <span class="error">&lt;source file=&quot;/images/vmx_20151102.0/images/metadata_usb.img&quot; /&gt;</span>
<span class="line-numbers"> 73</span>      <span class="error">&lt;target bus=&quot;usb&quot; dev=&quot;sda&quot; /&gt;</span>
<span class="line-numbers"> 74</span>    <span class="error">&lt;/disk&gt;</span>
<span class="line-numbers"> 75</span>
<span class="line-numbers"> 76</span>    <span class="error">&lt;controller index=&quot;0&quot; type=&quot;usb&quot;&gt;               #&lt;------pci device</span>
<span class="line-numbers"> 77</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x2&quot; slot=&quot;0x01&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 78</span>    <span class="error">&lt;/controller&gt;</span>
<span class="line-numbers"> 79</span>    <span class="error">&lt;controller index=&quot;0&quot; type=&quot;ide&quot;&gt;</span>
<span class="line-numbers"> 80</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x1&quot; slot=&quot;0x01&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 81</span>    <span class="error">&lt;/controller&gt;</span>
<span class="line-numbers"> 82</span>    <span class="error">&lt;controller index=&quot;0&quot; model=&quot;pci-root&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 83</span>
<span class="line-numbers"> 84</span>    <span class="error">&lt;interface type=&quot;bridge&quot;&gt;               #&lt;------generate bridge interface</span>
<span class="line-numbers"> 85</span>      <span class="error">&lt;source bridge=&quot;br-ext&quot; /&gt;            #&lt;------named vcp_ext-vmx1</span>
<span class="line-numbers"> 86</span>      <span class="error">&lt;target dev=&quot;vcp_ext-vmx1&quot; /&gt;         #&lt;------bound to br-ext bridge</span>
<span class="line-numbers"> 87</span>      <span class="error">&lt;model type=&quot;e1000&quot; /&gt;                #&lt;------soft emulation</span>
<span class="line-numbers"> 88</span>      <span class="error">&lt;mac address=&quot;02:12:DE:C0:DE:22&quot; /&gt;</span>
<span class="line-numbers"> 89</span>    <span class="error">&lt;/interface&gt;</span>
<span class="line-numbers"> 90</span>
<span class="line-numbers"> 91</span>    <span class="error">&lt;interface type=&quot;bridge&quot;&gt;               #&lt;------internal bridge</span>
<span class="line-numbers"> 92</span>      <span class="error">&lt;source bridge=&quot;br-int-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 93</span>      <span class="error">&lt;target dev=&quot;vcp_int-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 94</span>      <span class="error">&lt;model type=&quot;virtio&quot; /&gt;               #&lt;------virtio emulation</span>
<span class="line-numbers"> 95</span>    <span class="error">&lt;/interface&gt;</span>
<span class="line-numbers"> 96</span>
<span class="line-numbers"> 97</span>    <span class="error">&lt;serial type=&quot;tcp&quot;&gt;</span>
<span class="line-numbers"> 98</span>      <span class="error">&lt;source host=&quot;127.0.0.1&quot; mode=&quot;bind&quot; service=&quot;8896&quot; /&gt;</span>
<span class="line-numbers"> 99</span>      <span class="error">&lt;protocol type=&quot;telnet&quot; /&gt;</span>
<span class="line-numbers">100</span>      <span class="error">&lt;target port=&quot;0&quot; /&gt;</span>
<span class="line-numbers">101</span>    <span class="error">&lt;/serial&gt;</span>
<span class="line-numbers">102</span>
<span class="line-numbers">103</span>    <span class="error">&lt;console type=&quot;tcp&quot;&gt;</span>
<span class="line-numbers">104</span>      <span class="error">&lt;source host=&quot;127.0.0.1&quot; mode=&quot;bind&quot; service=&quot;8896&quot; /&gt;</span>
<span class="line-numbers">105</span>      <span class="error">&lt;protocol type=&quot;telnet&quot; /&gt;</span>
<span class="line-numbers">106</span>      <span class="error">&lt;target port=&quot;0&quot; type=&quot;serial&quot; /&gt;</span>
<span class="line-numbers">107</span>    <span class="error">&lt;/console&gt;</span>
<span class="line-numbers">108</span>
<span class="line-numbers">109</span>    <span class="error">&lt;input bus=&quot;usb&quot; type=&quot;tablet&quot; /&gt;</span>
<span class="line-numbers">110</span>    <span class="error">&lt;input bus=&quot;ps2&quot; type=&quot;mouse&quot; /&gt;</span>
<span class="line-numbers">111</span>    <span class="error">&lt;input bus=&quot;ps2&quot; type=&quot;keyboard&quot; /&gt;</span>
<span class="line-numbers">112</span>
<span class="line-numbers">113</span>    <span class="error">&lt;graphics autoport=&quot;yes&quot; listen=&quot;127.0.0.1&quot; port=&quot;-1&quot; type=&quot;vnc&quot;&gt;</span>
<span class="line-numbers">114</span>                                          <span class="comment">#&lt;------automatic vnc server port</span>
<span class="line-numbers">115</span>      <span class="error">&lt;listen address=&quot;127.0.0.1&quot; type=&quot;address&quot; /&gt;</span>
<span class="line-numbers">116</span>    <span class="error">&lt;/graphics&gt;</span>
<span class="line-numbers">117</span>
<span class="line-numbers">118</span>    <span class="error">&lt;sound model=&quot;ac97&quot;&gt;</span>
<span class="line-numbers">119</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x04&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers">120</span>    <span class="error">&lt;/sound&gt;</span>
<span class="line-numbers">121</span>
<span class="line-numbers">122</span>    <span class="error">&lt;video&gt;</span>
<span class="line-numbers">123</span>      <span class="error">&lt;model heads=&quot;1&quot; type=&quot;cirrus&quot; vram=&quot;9216&quot; /&gt;</span>
<span class="line-numbers">124</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x02&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers">125</span>    <span class="error">&lt;/video&gt;</span>
<span class="line-numbers">126</span>
<span class="line-numbers">127</span>    <span class="error">&lt;memballoon model=&quot;virtio&quot;&gt;            #&lt;------memory balloon location</span>
<span class="line-numbers">128</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x06&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers">129</span>    <span class="error">&lt;/memballoon&gt;</span>
<span class="line-numbers">130</span>
<span class="line-numbers">131</span>  <span class="error">&lt;/devices&gt;</span>
<span class="line-numbers">132</span><span class="error">&lt;/domain&gt;</span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_generated_vpfe_xml"><a class="anchor" href="#_generated_vpfe_xml"></a>generated vPFE xml</h3>
<div class="paragraph">
<p><code>build/vmx1/xml/vPFE-generated.xml</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers">  1</span><span class="error">ping@trinity:~$ cat xml.backup/vPFE-generated.xml</span>
<span class="line-numbers">  2</span><span class="error">&lt;domain type=&quot;kvm&quot;&gt;</span>
<span class="line-numbers">  3</span>
<span class="line-numbers">  4</span>  <span class="error">&lt;name&gt;vfp-vmx1&lt;/name&gt;</span>
<span class="line-numbers">  5</span>
<span class="line-numbers">  6</span>  <span class="error">&lt;memory unit=&quot;MB&quot;&gt;16384&lt;/memory&gt;</span>
<span class="line-numbers">  7</span>
<span class="line-numbers">  8</span>  <span class="error">&lt;memoryBacking&gt;</span>
<span class="line-numbers">  9</span>    <span class="error">&lt;nosharepages /&gt;</span>
<span class="line-numbers"> 10</span>    <span class="error">&lt;hugepages /&gt;</span>
<span class="line-numbers"> 11</span>  <span class="error">&lt;/memoryBacking&gt;</span>
<span class="line-numbers"> 12</span>
<span class="line-numbers"> 13</span>  <span class="error">&lt;vcpu placement=&quot;static&quot;&gt;4&lt;/vcpu&gt;</span>
<span class="line-numbers"> 14</span>
<span class="line-numbers"> 15</span>  <span class="error">&lt;numatune&gt;</span>
<span class="line-numbers"> 16</span>    <span class="error">&lt;memory mode=&quot;strict&quot; nodeset=&quot;1&quot; /&gt;</span>
<span class="line-numbers"> 17</span>  <span class="error">&lt;/numatune&gt;</span>
<span class="line-numbers"> 18</span>
<span class="line-numbers"> 19</span>  <span class="error">&lt;os&gt;</span>
<span class="line-numbers"> 20</span>    <span class="error">&lt;type arch=&quot;x86_64&quot; machine=&quot;pc-i440fx-trusty&quot;&gt;hvm&lt;/type&gt;</span>
<span class="line-numbers"> 21</span>    <span class="error">&lt;boot dev=&quot;hd&quot; /&gt;</span>
<span class="line-numbers"> 22</span>  <span class="error">&lt;/os&gt;</span>
<span class="line-numbers"> 23</span>
<span class="line-numbers"> 24</span>  <span class="error">&lt;features&gt;</span>
<span class="line-numbers"> 25</span>    <span class="error">&lt;acpi /&gt;</span>
<span class="line-numbers"> 26</span>  <span class="error">&lt;/features&gt;</span>
<span class="line-numbers"> 27</span>
<span class="line-numbers"> 28</span>  <span class="error">&lt;cpu mode=&quot;host-model&quot;&gt;</span>
<span class="line-numbers"> 29</span>    <span class="error">&lt;topology cores=&quot;4&quot; sockets=&quot;1&quot; threads=&quot;1&quot; /&gt;</span>
<span class="line-numbers"> 30</span>  <span class="error">&lt;/cpu&gt;</span>
<span class="line-numbers"> 31</span>
<span class="line-numbers"> 32</span>  <span class="error">&lt;clock offset=&quot;utc&quot; /&gt;</span>
<span class="line-numbers"> 33</span>  <span class="error">&lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;</span>
<span class="line-numbers"> 34</span>  <span class="error">&lt;on_reboot&gt;restart&lt;/on_reboot&gt;</span>
<span class="line-numbers"> 35</span>  <span class="error">&lt;on_crash&gt;restart&lt;/on_crash&gt;</span>
<span class="line-numbers"> 36</span>
<span class="line-numbers"> 37</span>  <span class="error">&lt;devices&gt;</span>
<span class="line-numbers"> 38</span>    <span class="error">&lt;emulator&gt;/usr/bin/qemu-system-x86_64&lt;/emulator&gt;</span>
<span class="line-numbers"> 39</span>
<span class="line-numbers"> 40</span>    <span class="error">&lt;disk device=&quot;disk&quot; type=&quot;file&quot;&gt;</span>
<span class="line-numbers"> 41</span>      <span class="error">&lt;driver cache=&quot;directsync&quot; name=&quot;qemu&quot; type=&quot;raw&quot; /&gt;</span>
<span class="line-numbers"> 42</span>      <span class="error">&lt;source file=&quot;/images/vmx_20151102.0/build/vmx1/images/vFPC-20151102.img&quot; /&gt;</span>
<span class="line-numbers"> 43</span>      <span class="error">&lt;target bus=&quot;ide&quot; dev=&quot;hda&quot; /&gt;</span>
<span class="line-numbers"> 44</span>    <span class="error">&lt;/disk&gt;</span>
<span class="line-numbers"> 45</span>
<span class="line-numbers"> 46</span>    <span class="error">&lt;controller index=&quot;0&quot; model=&quot;pci-root&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 47</span>
<span class="line-numbers"> 48</span>    <span class="error">&lt;interface type=&quot;bridge&quot;&gt;</span>
<span class="line-numbers"> 49</span>      <span class="error">&lt;source bridge=&quot;br-ext&quot; /&gt;</span>
<span class="line-numbers"> 50</span>      <span class="error">&lt;target dev=&quot;vfp_ext-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 51</span>      <span class="error">&lt;model type=&quot;virtio&quot; /&gt;</span>
<span class="line-numbers"> 52</span>      <span class="error">&lt;alias name=&quot;net0&quot; /&gt;</span>
<span class="line-numbers"> 53</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x03&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 54</span>      <span class="error">&lt;mac address=&quot;02:12:DE:C0:DE:23&quot; /&gt;</span>
<span class="line-numbers"> 55</span>    <span class="error">&lt;/interface&gt;</span>
<span class="line-numbers"> 56</span>
<span class="line-numbers"> 57</span>    <span class="error">&lt;interface type=&quot;bridge&quot;&gt;</span>
<span class="line-numbers"> 58</span>      <span class="error">&lt;source bridge=&quot;br-int-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 59</span>      <span class="error">&lt;target dev=&quot;vfp_int-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 60</span>      <span class="error">&lt;model type=&quot;virtio&quot; /&gt;</span>
<span class="line-numbers"> 61</span>      <span class="error">&lt;alias name=&quot;net0&quot; /&gt;</span>
<span class="line-numbers"> 62</span>      <span class="error">&lt;address bus=&quot;0x00&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x04&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 63</span>    <span class="error">&lt;/interface&gt;</span>
<span class="line-numbers"> 64</span>
<span class="line-numbers"> 65</span>    <span class="error">&lt;serial type=&quot;tcp&quot;&gt;</span>
<span class="line-numbers"> 66</span>      <span class="error">&lt;source host=&quot;127.0.0.1&quot; mode=&quot;bind&quot; service=&quot;8897&quot; /&gt;</span>
<span class="line-numbers"> 67</span>      <span class="error">&lt;protocol type=&quot;telnet&quot; /&gt;</span>
<span class="line-numbers"> 68</span>      <span class="error">&lt;target port=&quot;0&quot; /&gt;</span>
<span class="line-numbers"> 69</span>    <span class="error">&lt;/serial&gt;</span>
<span class="line-numbers"> 70</span>
<span class="line-numbers"> 71</span>    <span class="error">&lt;console type=&quot;tcp&quot;&gt;</span>
<span class="line-numbers"> 72</span>      <span class="error">&lt;source host=&quot;127.0.0.1&quot; mode=&quot;bind&quot; service=&quot;8897&quot; /&gt;</span>
<span class="line-numbers"> 73</span>      <span class="error">&lt;protocol type=&quot;telnet&quot; /&gt;</span>
<span class="line-numbers"> 74</span>      <span class="error">&lt;target port=&quot;0&quot; type=&quot;serial&quot; /&gt;</span>
<span class="line-numbers"> 75</span>    <span class="error">&lt;/console&gt;</span>
<span class="line-numbers"> 76</span>
<span class="line-numbers"> 77</span>    <span class="error">&lt;input bus=&quot;usb&quot; type=&quot;tablet&quot; /&gt;</span>
<span class="line-numbers"> 78</span>    <span class="error">&lt;input bus=&quot;ps2&quot; type=&quot;mouse&quot; /&gt;</span>
<span class="line-numbers"> 79</span>    <span class="error">&lt;input bus=&quot;ps2&quot; type=&quot;keyboard&quot; /&gt;</span>
<span class="line-numbers"> 80</span>    <span class="error">&lt;graphics autoport=&quot;yes&quot; listen=&quot;127.0.0.1&quot; port=&quot;-1&quot; type=&quot;vnc&quot;&gt;</span>
<span class="line-numbers"> 81</span>      <span class="error">&lt;listen address=&quot;127.0.0.1&quot; type=&quot;address&quot; /&gt;</span>
<span class="line-numbers"> 82</span>    <span class="error">&lt;/graphics&gt;</span>
<span class="line-numbers"> 83</span>    <span class="error">&lt;sound model=&quot;ac97&quot;&gt;</span>
<span class="line-numbers"> 84</span>    <span class="error">&lt;/sound&gt;</span>
<span class="line-numbers"> 85</span>    <span class="error">&lt;video&gt;</span>
<span class="line-numbers"> 86</span>      <span class="error">&lt;model heads=&quot;1&quot; type=&quot;cirrus&quot; vram=&quot;9216&quot; /&gt;</span>
<span class="line-numbers"> 87</span>    <span class="error">&lt;/video&gt;</span>
<span class="line-numbers"> 88</span>
<span class="line-numbers"> 89</span>    <span class="error">&lt;memballoon model=&quot;virtio&quot;&gt;</span>
<span class="line-numbers"> 90</span>    <span class="error">&lt;/memballoon&gt;</span>
<span class="line-numbers"> 91</span>
<span class="line-numbers"> 92</span>    <span class="error">&lt;interface managed=&quot;yes&quot; type=&quot;hostdev&quot;&gt;        #&lt;------VT-d SR-IOV VF assignment</span>
<span class="line-numbers"> 93</span>      <span class="error">&lt;mac address=&quot;02:16:0A:0E:FF:31&quot; /&gt;           #&lt;------MAC appeared to VM</span>
<span class="line-numbers"> 94</span>      <span class="error">&lt;source&gt;                                      #&lt;------VF to be assigned</span>
<span class="line-numbers"> 95</span>        <span class="error">&lt;address bus=&quot;0x23&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x10&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers"> 96</span>      <span class="error">&lt;/source&gt;                                     #&lt;------23:10.0 is p3p1 vf0</span>
<span class="line-numbers"> 97</span>    <span class="error">&lt;/interface&gt;</span>
<span class="line-numbers"> 98</span>
<span class="line-numbers"> 99</span>    <span class="error">&lt;interface managed=&quot;yes&quot; type=&quot;hostdev&quot;&gt;</span>
<span class="line-numbers">100</span>      <span class="error">&lt;mac address=&quot;02:16:0A:0E:FF:32&quot; /&gt;</span>
<span class="line-numbers">101</span>      <span class="error">&lt;source&gt;</span>
<span class="line-numbers">102</span>        <span class="error">&lt;address bus=&quot;0x06&quot; domain=&quot;0x0000&quot; function=&quot;0x0&quot; slot=&quot;0x10&quot; type=&quot;pci&quot; /&gt;</span>
<span class="line-numbers">103</span>      <span class="error">&lt;/source&gt;                                     #&lt;------06:10.0 is p2p1 vf0</span>
<span class="line-numbers">104</span>    <span class="error">&lt;/interface&gt;</span>
<span class="line-numbers">105</span>  <span class="error">&lt;/devices&gt;</span>
<span class="line-numbers">106</span>
<span class="line-numbers">107</span><span class="error">&lt;/domain&gt;</span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_generated_virtual_network_xml"><a class="anchor" href="#_generated_virtual_network_xml"></a>generated virtual network XML</h3>
<div class="listingblock">
<div class="title">br-ext-generated.xml:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="error">&lt;network&gt;</span>
<span class="line-numbers"> 2</span>  <span class="error">&lt;name&gt;br-ext&lt;/name&gt;</span>
<span class="line-numbers"> 3</span>  <span class="error">&lt;forward mode=&quot;route&quot; /&gt;</span>
<span class="line-numbers"> 4</span>  <span class="error">&lt;bridge delay=&quot;0&quot; name=&quot;br-ext&quot; stp=&quot;on&quot; /&gt;</span>
<span class="line-numbers"> 5</span>  <span class="error">&lt;mac address=&quot;52:54:00:9f:a0:77&quot; /&gt;</span>
<span class="line-numbers"> 6</span>  <span class="error">&lt;ip address=&quot;10.85.4.17&quot; netmask=&quot;255.255.255.128&quot;&gt;</span>
<span class="line-numbers"> 7</span>    <span class="error">&lt;dhcp&gt;</span>
<span class="line-numbers"> 8</span>      <span class="error">&lt;host ip=&quot;10.85.4.105&quot; mac=&quot;02:04:17:01:01:01&quot; name=&quot;vcp-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 9</span>      <span class="error">&lt;host ip=&quot;10.85.4.106&quot; mac=&quot;02:04:17:01:01:02&quot; name=&quot;vfp-vmx1&quot; /&gt;</span>
<span class="line-numbers">10</span>    <span class="error">&lt;/dhcp&gt;</span>
<span class="line-numbers">11</span>  <span class="error">&lt;/ip&gt;</span>
<span class="line-numbers">12</span><span class="error">&lt;/network&gt;</span></code></pre>
</div>
</div>
<div class="listingblock">
<div class="title">br-int-generated.xml:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers">1</span><span class="error">&lt;network&gt;</span>
<span class="line-numbers">2</span>  <span class="error">&lt;name&gt;br-int-vmx1&lt;/name&gt;</span>
<span class="line-numbers">3</span>  <span class="error">&lt;bridge delay=&quot;0&quot; name=&quot;br-int-vmx1&quot; stp=&quot;on&quot; /&gt;</span>
<span class="line-numbers">4</span><span class="error">&lt;/network&gt;</span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="shell-files"><a class="anchor" href="#shell-files"></a>generated shell scripts</h3>
<div class="listingblock">
<div class="title"><code>build/vmx1/xml/cpu_affinitize.sh</code>:</div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers">1</span>virsh vcpupin vfp-vmx1 0 11
<span class="line-numbers">2</span>virsh vcpupin vfp-vmx1 1 12
<span class="line-numbers">3</span>virsh vcpupin vfp-vmx1 2 13
<span class="line-numbers">4</span>virsh vcpupin vfp-vmx1 3 8
<span class="line-numbers">5</span>virsh vcpupin vfp-vmx1 4 9
<span class="line-numbers">6</span>virsh vcpupin vcp-vmx1 0 15
<span class="line-numbers">7</span>virsh emulatorpin vcp-vmx1 0
<span class="line-numbers">8</span>virsh emulatorpin vfp-vmx1 0</code></pre>
</div>
</div>
<div class="listingblock">
<div class="title"><code>build/vmx1/xml/vfconfig-generated.sh</code></div>
<div class="content">
<pre class="CodeRay highlight"><code data-lang="bash"><span class="line-numbers"> 1</span>#Handling interface p3p1
<span class="line-numbers"> 2</span>ifconfig p3p1 up
<span class="line-numbers"> 3</span>sleep 2
<span class="line-numbers"> 4</span>ifconfig p3p1 promisc
<span class="line-numbers"> 5</span>ifconfig p3p1 allmulti
<span class="line-numbers"> 6</span>ifconfig p3p1 mtu 2000
<span class="line-numbers"> 7</span>ip link set p3p1 vf 0 mac 02:04:17:01:02:01
<span class="line-numbers"> 8</span>ip link set p3p1 vf 0 rate 10000
<span class="line-numbers"> 9</span>echo 0000:23:10.0 &gt; /sys/bus/pci/devices/0000:23:10.0/driver/unbind
<span class="line-numbers">10</span>echo 0000:23:10.0 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">11</span>ip link set p3p1 vf 0 spoofchk off
<span class="line-numbers">12</span>
<span class="line-numbers">13</span>#Handling interface p2p1
<span class="line-numbers">14</span>ifconfig p2p1 up
<span class="line-numbers">15</span>sleep 2
<span class="line-numbers">16</span>ifconfig p2p1 promisc
<span class="line-numbers">17</span>ifconfig p2p1 allmulti
<span class="line-numbers">18</span>ifconfig p2p1 mtu 2000
<span class="line-numbers">19</span>ip link set p2p1 vf 0 mac 02:04:17:01:02:02
<span class="line-numbers">20</span>ip link set p2p1 vf 0 rate 10000
<span class="line-numbers">21</span>echo 0000:06:10.0 &gt; /sys/bus/pci/devices/0000:06:10.0/driver/unbind
<span class="line-numbers">22</span>echo 0000:06:10.0 &gt;&gt; /sys/bus/pci/drivers/pci-stub/bind
<span class="line-numbers">23</span>ip link set p2p1 vf 0 spoofchk off</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_generated_files_for_virtio"><a class="anchor" href="#_generated_files_for_virtio"></a>generated files for VIRTIO</h3>
<div class="paragraph">
<p>comparing with SRIOV, the generated files for VIRTIO remains the same except:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>the "interface" part for vPFE VM now looks:</p>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="yaml"><span class="line-numbers"> 1</span><span class="head"><span class="head">...</span></span><span class="error">...</span>
<span class="line-numbers"> 2</span><span class="error">&lt;interface type=&quot;network&quot;&gt;</span>
<span class="line-numbers"> 3</span>  <span class="error">&lt;mac address=&quot;02:04:17:01:02:01&quot; /&gt;</span>
<span class="line-numbers"> 4</span>  <span class="error">&lt;source network=&quot;default&quot; /&gt;</span>
<span class="line-numbers"> 5</span>  <span class="error">&lt;model type=&quot;virtio&quot; /&gt;</span>
<span class="line-numbers"> 6</span>  <span class="error">&lt;target dev=&quot;ge-0.0.0-vmx1&quot; /&gt;</span>
<span class="line-numbers"> 7</span><span class="error">&lt;/interface&gt;</span>
<span class="line-numbers"> 8</span><span class="error">&lt;interface type=&quot;network&quot;&gt;</span>
<span class="line-numbers"> 9</span>  <span class="error">&lt;mac address=&quot;02:04:17:01:02:02&quot; /&gt;</span>
<span class="line-numbers">10</span>  <span class="error">&lt;source network=&quot;default&quot; /&gt;</span>
<span class="line-numbers">11</span>  <span class="error">&lt;model type=&quot;virtio&quot; /&gt;</span>
<span class="line-numbers">12</span>  <span class="error">&lt;target dev=&quot;ge-0.0.1-vmx1&quot; /&gt;</span>
<span class="line-numbers">13</span><span class="error">&lt;/interface&gt;</span></code></pre>
</div>
</div>
</li>
<li>
<p>since there is no concept of VF for VIRTIO virtual interface, there is no
script generated for VF configuration.</p>
</li>
</ol>
</div>
</div>
<div class="sect2">
<h3 id="_dumpxml_vcp_vmx1"><a class="anchor" href="#_dumpxml_vcp_vmx1"></a>dumpxml vcp-vmx1</h3>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers">  1</span>ping@trinity:/virtualization/vmx1$ cat vRE-generated-all.xml
<span class="line-numbers">  2</span><span class="tag">&lt;domain</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">kvm</span><span class="delimiter">'</span></span> <span class="attribute-name">id</span>=<span class="string"><span class="delimiter">'</span><span class="content">2</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">  3</span>  <span class="tag">&lt;name&gt;</span>vcp-vmx1<span class="tag">&lt;/name&gt;</span>
<span class="line-numbers">  4</span>  <span class="tag">&lt;uuid&gt;</span>956ce015-cf26-4752-8679-6925f512870c<span class="tag">&lt;/uuid&gt;</span>
<span class="line-numbers">  5</span>  <span class="tag">&lt;memory</span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">KiB</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>2000896<span class="tag">&lt;/memory&gt;</span>
<span class="line-numbers">  6</span>  <span class="tag">&lt;currentMemory</span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">KiB</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>2000000<span class="tag">&lt;/currentMemory&gt;</span>
<span class="line-numbers">  7</span>  <span class="tag">&lt;vcpu</span> <span class="attribute-name">placement</span>=<span class="string"><span class="delimiter">'</span><span class="content">static</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>1<span class="tag">&lt;/vcpu&gt;</span>
<span class="line-numbers">  8</span>  <span class="tag">&lt;cputune&gt;</span>
<span class="line-numbers">  9</span>    <span class="tag">&lt;vcpupin</span> <span class="attribute-name">vcpu</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">15</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 10</span>    <span class="tag">&lt;emulatorpin</span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 11</span>  <span class="tag">&lt;/cputune&gt;</span>
<span class="line-numbers"> 12</span>  <span class="tag">&lt;resource&gt;</span>
<span class="line-numbers"> 13</span>    <span class="tag">&lt;partition&gt;</span>/machine<span class="tag">&lt;/partition&gt;</span>
<span class="line-numbers"> 14</span>  <span class="tag">&lt;/resource&gt;</span>
<span class="line-numbers"> 15</span>    <span class="tag">&lt;sysinfo</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">smbios</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 16</span>      <span class="tag">&lt;bios&gt;</span>
<span class="line-numbers"> 17</span>        <span class="tag">&lt;entry</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">vendor</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>Juniper<span class="tag">&lt;/entry&gt;</span>
<span class="line-numbers"> 18</span>      <span class="tag">&lt;/bios&gt;</span>
<span class="line-numbers"> 19</span>      <span class="tag">&lt;system&gt;</span>
<span class="line-numbers"> 20</span>        <span class="tag">&lt;entry</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">manufacturer</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>VMX<span class="tag">&lt;/entry&gt;</span>
<span class="line-numbers"> 21</span>        <span class="tag">&lt;entry</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">product</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>VM-vcp_vmx1-161-re-0<span class="tag">&lt;/entry&gt;</span>
<span class="line-numbers"> 22</span>        <span class="tag">&lt;entry</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">version</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>0.1.0<span class="tag">&lt;/entry&gt;</span>
<span class="line-numbers"> 23</span>      <span class="tag">&lt;/system&gt;</span>
<span class="line-numbers"> 24</span>    <span class="tag">&lt;/sysinfo&gt;</span>
<span class="line-numbers"> 25</span>  <span class="tag">&lt;os&gt;</span>
<span class="line-numbers"> 26</span>    <span class="tag">&lt;type</span> <span class="attribute-name">arch</span>=<span class="string"><span class="delimiter">'</span><span class="content">x86_64</span><span class="delimiter">'</span></span> <span class="attribute-name">machine</span>=<span class="string"><span class="delimiter">'</span><span class="content">pc-0.13</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>hvm<span class="tag">&lt;/type&gt;</span>
<span class="line-numbers"> 27</span>    <span class="tag">&lt;boot</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">hd</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 28</span>    <span class="tag">&lt;smbios</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">sysinfo</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 29</span>  <span class="tag">&lt;/os&gt;</span>
<span class="line-numbers"> 30</span>  <span class="tag">&lt;features&gt;</span>
<span class="line-numbers"> 31</span>    <span class="tag">&lt;acpi</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 32</span>    <span class="tag">&lt;apic</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 33</span>    <span class="tag">&lt;pae</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 34</span>  <span class="tag">&lt;/features&gt;</span>
<span class="line-numbers"> 35</span>  <span class="tag">&lt;cpu</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">host-model</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 36</span>    <span class="tag">&lt;model</span> <span class="attribute-name">fallback</span>=<span class="string"><span class="delimiter">'</span><span class="content">allow</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 37</span>    <span class="tag">&lt;topology</span> <span class="attribute-name">sockets</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span> <span class="attribute-name">cores</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span> <span class="attribute-name">threads</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 38</span>  <span class="tag">&lt;/cpu&gt;</span>
<span class="line-numbers"> 39</span>  <span class="tag">&lt;clock</span> <span class="attribute-name">offset</span>=<span class="string"><span class="delimiter">'</span><span class="content">utc</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 40</span>  <span class="tag">&lt;on_poweroff&gt;</span>destroy<span class="tag">&lt;/on_poweroff&gt;</span>
<span class="line-numbers"> 41</span>  <span class="tag">&lt;on_reboot&gt;</span>restart<span class="tag">&lt;/on_reboot&gt;</span>
<span class="line-numbers"> 42</span>  <span class="tag">&lt;on_crash&gt;</span>restart<span class="tag">&lt;/on_crash&gt;</span>
<span class="line-numbers"> 43</span>  <span class="tag">&lt;devices&gt;</span>
<span class="line-numbers"> 44</span>    <span class="tag">&lt;emulator&gt;</span>/usr/bin/qemu-system-x86_64<span class="tag">&lt;/emulator&gt;</span>
<span class="line-numbers"> 45</span>    <span class="tag">&lt;disk</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">file</span><span class="delimiter">'</span></span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">'</span><span class="content">disk</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 46</span>      <span class="tag">&lt;driver</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">qemu</span><span class="delimiter">'</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">qcow2</span><span class="delimiter">'</span></span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">'</span><span class="content">directsync</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 47</span>      <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">'</span><span class="content">/virtualization/images/vmx_20151102.0/build/vmx1/images/jinstall64-vmx-15.1F-20151104.0-domestic.img</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 48</span>      <span class="tag">&lt;backingStore</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 49</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">hda</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 50</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide0-0-0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 51</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">drive</span><span class="delimiter">'</span></span> <span class="attribute-name">controller</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">target</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 52</span>    <span class="tag">&lt;/disk&gt;</span>
<span class="line-numbers"> 53</span>    <span class="tag">&lt;disk</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">file</span><span class="delimiter">'</span></span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">'</span><span class="content">disk</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 54</span>      <span class="tag">&lt;driver</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">qemu</span><span class="delimiter">'</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">qcow2</span><span class="delimiter">'</span></span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">'</span><span class="content">directsync</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 55</span>      <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">'</span><span class="content">/virtualization/images/vmx_20151102.0/build/vmx1/images/vmxhdd.img</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 56</span>      <span class="tag">&lt;backingStore</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 57</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">hdb</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 58</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide0-0-1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 59</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">drive</span><span class="delimiter">'</span></span> <span class="attribute-name">controller</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">target</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 60</span>    <span class="tag">&lt;/disk&gt;</span>
<span class="line-numbers"> 61</span>    <span class="tag">&lt;disk</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">file</span><span class="delimiter">'</span></span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">'</span><span class="content">disk</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 62</span>      <span class="tag">&lt;driver</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">qemu</span><span class="delimiter">'</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">raw</span><span class="delimiter">'</span></span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">'</span><span class="content">directsync</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 63</span>      <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">'</span><span class="content">/virtualization/images/vmx_20151102.0/images/metadata_usb.img</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 64</span>      <span class="tag">&lt;backingStore</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 65</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">sda</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 66</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb-disk0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 67</span>    <span class="tag">&lt;/disk&gt;</span>
<span class="line-numbers"> 68</span>    <span class="tag">&lt;controller</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb</span><span class="delimiter">'</span></span> <span class="attribute-name">index</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 69</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 70</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x01</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 71</span>    <span class="tag">&lt;/controller&gt;</span>
<span class="line-numbers"> 72</span>    <span class="tag">&lt;controller</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide</span><span class="delimiter">'</span></span> <span class="attribute-name">index</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 73</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 74</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x01</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 75</span>    <span class="tag">&lt;/controller&gt;</span>
<span class="line-numbers"> 76</span>    <span class="tag">&lt;controller</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">index</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">model</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci-root</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 77</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci.0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 78</span>    <span class="tag">&lt;/controller&gt;</span>
<span class="line-numbers"> 79</span>    <span class="tag">&lt;interface</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">bridge</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 80</span>      <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">02:04:17:01:01:01</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 81</span>      <span class="tag">&lt;source</span> <span class="attribute-name">bridge</span>=<span class="string"><span class="delimiter">'</span><span class="content">br-ext</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 82</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">vcp_ext-vmx1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 83</span>      <span class="tag">&lt;model</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">e1000</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 84</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">net0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 85</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x03</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 86</span>    <span class="tag">&lt;/interface&gt;</span>
<span class="line-numbers"> 87</span>    <span class="tag">&lt;interface</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">bridge</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 88</span>      <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">52:54:00:7a:91:1a</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 89</span>      <span class="tag">&lt;source</span> <span class="attribute-name">bridge</span>=<span class="string"><span class="delimiter">'</span><span class="content">br-int-vmx1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 90</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">vcp_int-vmx1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 91</span>      <span class="tag">&lt;model</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">virtio</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 92</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">net1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 93</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x05</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 94</span>    <span class="tag">&lt;/interface&gt;</span>
<span class="line-numbers"> 95</span>    <span class="tag">&lt;serial</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">tcp</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 96</span>      <span class="tag">&lt;source</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">bind</span><span class="delimiter">'</span></span> <span class="attribute-name">host</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span> <span class="attribute-name">service</span>=<span class="string"><span class="delimiter">'</span><span class="content">8816</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 97</span>      <span class="tag">&lt;protocol</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">telnet</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 98</span>      <span class="tag">&lt;target</span> <span class="attribute-name">port</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 99</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">serial0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">100</span>    <span class="tag">&lt;/serial&gt;</span>
<span class="line-numbers">101</span>    <span class="tag">&lt;console</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">tcp</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">102</span>      <span class="tag">&lt;source</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">bind</span><span class="delimiter">'</span></span> <span class="attribute-name">host</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span> <span class="attribute-name">service</span>=<span class="string"><span class="delimiter">'</span><span class="content">8816</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">103</span>      <span class="tag">&lt;protocol</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">telnet</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">104</span>      <span class="tag">&lt;target</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">serial</span><span class="delimiter">'</span></span> <span class="attribute-name">port</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">105</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">serial0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">106</span>    <span class="tag">&lt;/console&gt;</span>
<span class="line-numbers">107</span>    <span class="tag">&lt;input</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">tablet</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">108</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">input0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">109</span>    <span class="tag">&lt;/input&gt;</span>
<span class="line-numbers">110</span>    <span class="tag">&lt;input</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">mouse</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ps2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">111</span>    <span class="tag">&lt;input</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">keyboard</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ps2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">112</span>    <span class="tag">&lt;graphics</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">vnc</span><span class="delimiter">'</span></span> <span class="attribute-name">port</span>=<span class="string"><span class="delimiter">'</span><span class="content">5900</span><span class="delimiter">'</span></span> <span class="attribute-name">autoport</span>=<span class="string"><span class="delimiter">'</span><span class="content">yes</span><span class="delimiter">'</span></span> <span class="attribute-name">listen</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">113</span>      <span class="tag">&lt;listen</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">address</span><span class="delimiter">'</span></span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">114</span>    <span class="tag">&lt;/graphics&gt;</span>
<span class="line-numbers">115</span>    <span class="tag">&lt;sound</span> <span class="attribute-name">model</span>=<span class="string"><span class="delimiter">'</span><span class="content">ac97</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">116</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">sound0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">117</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x04</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">118</span>    <span class="tag">&lt;/sound&gt;</span>
<span class="line-numbers">119</span>    <span class="tag">&lt;video&gt;</span>
<span class="line-numbers">120</span>      <span class="tag">&lt;model</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">cirrus</span><span class="delimiter">'</span></span> <span class="attribute-name">vram</span>=<span class="string"><span class="delimiter">'</span><span class="content">9216</span><span class="delimiter">'</span></span> <span class="attribute-name">heads</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">121</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">video0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">122</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x02</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">123</span>    <span class="tag">&lt;/video&gt;</span>
<span class="line-numbers">124</span>    <span class="tag">&lt;memballoon</span> <span class="attribute-name">model</span>=<span class="string"><span class="delimiter">'</span><span class="content">virtio</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">125</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">balloon0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">126</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x06</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">127</span>    <span class="tag">&lt;/memballoon&gt;</span>
<span class="line-numbers">128</span>  <span class="tag">&lt;/devices&gt;</span>
<span class="line-numbers">129</span><span class="tag">&lt;/domain&gt;</span></code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_dumpxml_vfp_vmx1"><a class="anchor" href="#_dumpxml_vfp_vmx1"></a>dumpxml vfp-vmx1</h3>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><span class="line-numbers">  1</span>ping@trinity:/virtualization/vmx1$ cat vPFE-generated-all.xml
<span class="line-numbers">  2</span><span class="tag">&lt;domain</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">kvm</span><span class="delimiter">'</span></span> <span class="attribute-name">id</span>=<span class="string"><span class="delimiter">'</span><span class="content">3</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">  3</span>  <span class="tag">&lt;name&gt;</span>vfp-vmx1<span class="tag">&lt;/name&gt;</span>
<span class="line-numbers">  4</span>  <span class="tag">&lt;uuid&gt;</span>399d395f-d583-42e5-ace9-86118b346565<span class="tag">&lt;/uuid&gt;</span>
<span class="line-numbers">  5</span>  <span class="tag">&lt;memory</span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">KiB</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>16000000<span class="tag">&lt;/memory&gt;</span>
<span class="line-numbers">  6</span>  <span class="tag">&lt;currentMemory</span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">KiB</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>16000000<span class="tag">&lt;/currentMemory&gt;</span>
<span class="line-numbers">  7</span>  <span class="tag">&lt;memoryBacking&gt;</span>
<span class="line-numbers">  8</span>    <span class="tag">&lt;hugepages</span><span class="tag">/&gt;</span>
<span class="line-numbers">  9</span>    <span class="tag">&lt;nosharepages</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 10</span>  <span class="tag">&lt;/memoryBacking&gt;</span>
<span class="line-numbers"> 11</span>  <span class="tag">&lt;vcpu</span> <span class="attribute-name">placement</span>=<span class="string"><span class="delimiter">'</span><span class="content">static</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>4<span class="tag">&lt;/vcpu&gt;</span>
<span class="line-numbers"> 12</span>  <span class="tag">&lt;cputune&gt;</span>
<span class="line-numbers"> 13</span>    <span class="tag">&lt;vcpupin</span> <span class="attribute-name">vcpu</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">11</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 14</span>    <span class="tag">&lt;vcpupin</span> <span class="attribute-name">vcpu</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">12</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 15</span>    <span class="tag">&lt;vcpupin</span> <span class="attribute-name">vcpu</span>=<span class="string"><span class="delimiter">'</span><span class="content">2</span><span class="delimiter">'</span></span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">13</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 16</span>    <span class="tag">&lt;vcpupin</span> <span class="attribute-name">vcpu</span>=<span class="string"><span class="delimiter">'</span><span class="content">3</span><span class="delimiter">'</span></span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">8</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 17</span>    <span class="tag">&lt;emulatorpin</span> <span class="attribute-name">cpuset</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 18</span>  <span class="tag">&lt;/cputune&gt;</span>
<span class="line-numbers"> 19</span>  <span class="tag">&lt;numatune&gt;</span>
<span class="line-numbers"> 20</span>    <span class="tag">&lt;memory</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">strict</span><span class="delimiter">'</span></span> <span class="attribute-name">nodeset</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 21</span>  <span class="tag">&lt;/numatune&gt;</span>
<span class="line-numbers"> 22</span>  <span class="tag">&lt;resource&gt;</span>
<span class="line-numbers"> 23</span>    <span class="tag">&lt;partition&gt;</span>/machine<span class="tag">&lt;/partition&gt;</span>
<span class="line-numbers"> 24</span>  <span class="tag">&lt;/resource&gt;</span>
<span class="line-numbers"> 25</span>  <span class="tag">&lt;os&gt;</span>
<span class="line-numbers"> 26</span>    <span class="tag">&lt;type</span> <span class="attribute-name">arch</span>=<span class="string"><span class="delimiter">'</span><span class="content">x86_64</span><span class="delimiter">'</span></span> <span class="attribute-name">machine</span>=<span class="string"><span class="delimiter">'</span><span class="content">pc-i440fx-trusty</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>hvm<span class="tag">&lt;/type&gt;</span>
<span class="line-numbers"> 27</span>    <span class="tag">&lt;boot</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">hd</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 28</span>  <span class="tag">&lt;/os&gt;</span>
<span class="line-numbers"> 29</span>  <span class="tag">&lt;features&gt;</span>
<span class="line-numbers"> 30</span>    <span class="tag">&lt;acpi</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 31</span>  <span class="tag">&lt;/features&gt;</span>
<span class="line-numbers"> 32</span>  <span class="tag">&lt;cpu</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">host-model</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 33</span>    <span class="tag">&lt;model</span> <span class="attribute-name">fallback</span>=<span class="string"><span class="delimiter">'</span><span class="content">allow</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 34</span>    <span class="tag">&lt;topology</span> <span class="attribute-name">sockets</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span> <span class="attribute-name">cores</span>=<span class="string"><span class="delimiter">'</span><span class="content">4</span><span class="delimiter">'</span></span> <span class="attribute-name">threads</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 35</span>  <span class="tag">&lt;/cpu&gt;</span>
<span class="line-numbers"> 36</span>  <span class="tag">&lt;clock</span> <span class="attribute-name">offset</span>=<span class="string"><span class="delimiter">'</span><span class="content">utc</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 37</span>  <span class="tag">&lt;on_poweroff&gt;</span>destroy<span class="tag">&lt;/on_poweroff&gt;</span>
<span class="line-numbers"> 38</span>  <span class="tag">&lt;on_reboot&gt;</span>restart<span class="tag">&lt;/on_reboot&gt;</span>
<span class="line-numbers"> 39</span>  <span class="tag">&lt;on_crash&gt;</span>restart<span class="tag">&lt;/on_crash&gt;</span>
<span class="line-numbers"> 40</span>  <span class="tag">&lt;devices&gt;</span>
<span class="line-numbers"> 41</span>    <span class="tag">&lt;emulator&gt;</span>/usr/bin/qemu-system-x86_64<span class="tag">&lt;/emulator&gt;</span>
<span class="line-numbers"> 42</span>    <span class="tag">&lt;disk</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">file</span><span class="delimiter">'</span></span> <span class="attribute-name">device</span>=<span class="string"><span class="delimiter">'</span><span class="content">disk</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 43</span>      <span class="tag">&lt;driver</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">qemu</span><span class="delimiter">'</span></span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">raw</span><span class="delimiter">'</span></span> <span class="attribute-name">cache</span>=<span class="string"><span class="delimiter">'</span><span class="content">directsync</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 44</span>      <span class="tag">&lt;source</span> <span class="attribute-name">file</span>=<span class="string"><span class="delimiter">'</span><span class="content">/virtualization/images/vmx_20151102.0/build/vmx1/images/vFPC-20151102.img</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 45</span>      <span class="tag">&lt;backingStore</span><span class="tag">/&gt;</span>
<span class="line-numbers"> 46</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">hda</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 47</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide0-0-0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 48</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">drive</span><span class="delimiter">'</span></span> <span class="attribute-name">controller</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">target</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">unit</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 49</span>    <span class="tag">&lt;/disk&gt;</span>
<span class="line-numbers"> 50</span>    <span class="tag">&lt;controller</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">index</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span> <span class="attribute-name">model</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci-root</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 51</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci.0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 52</span>    <span class="tag">&lt;/controller&gt;</span>
<span class="line-numbers"> 53</span>    <span class="tag">&lt;controller</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb</span><span class="delimiter">'</span></span> <span class="attribute-name">index</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 54</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 55</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x01</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 56</span>    <span class="tag">&lt;/controller&gt;</span>
<span class="line-numbers"> 57</span>    <span class="tag">&lt;controller</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide</span><span class="delimiter">'</span></span> <span class="attribute-name">index</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 58</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">ide0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 59</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x01</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 60</span>    <span class="tag">&lt;/controller&gt;</span>
<span class="line-numbers"> 61</span>    <span class="tag">&lt;interface</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">bridge</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 62</span>      <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">02:04:17:01:01:02</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 63</span>      <span class="tag">&lt;source</span> <span class="attribute-name">bridge</span>=<span class="string"><span class="delimiter">'</span><span class="content">br-ext</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 64</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">vfp_ext-vmx1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 65</span>      <span class="tag">&lt;model</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">virtio</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 66</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">net0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 67</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x03</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 68</span>    <span class="tag">&lt;/interface&gt;</span>
<span class="line-numbers"> 69</span>    <span class="tag">&lt;interface</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">bridge</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 70</span>      <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">52:54:00:9c:b3:f2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 71</span>      <span class="tag">&lt;source</span> <span class="attribute-name">bridge</span>=<span class="string"><span class="delimiter">'</span><span class="content">br-int-vmx1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 72</span>      <span class="tag">&lt;target</span> <span class="attribute-name">dev</span>=<span class="string"><span class="delimiter">'</span><span class="content">vfp_int-vmx1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 73</span>      <span class="tag">&lt;model</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">virtio</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 74</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">net1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 75</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x04</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 76</span>    <span class="tag">&lt;/interface&gt;</span>
<span class="line-numbers"> 77</span>    <span class="tag">&lt;interface</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">hostdev</span><span class="delimiter">'</span></span> <span class="attribute-name">managed</span>=<span class="string"><span class="delimiter">'</span><span class="content">yes</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 78</span>      <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">02:04:17:01:02:01</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 79</span>      <span class="tag">&lt;driver</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">kvm</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 80</span>      <span class="tag">&lt;source&gt;</span>
<span class="line-numbers"> 81</span>        <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x23</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x10</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 82</span>      <span class="tag">&lt;/source&gt;</span>
<span class="line-numbers"> 83</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">hostdev0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 84</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x06</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 85</span>    <span class="tag">&lt;/interface&gt;</span>
<span class="line-numbers"> 86</span>    <span class="tag">&lt;interface</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">hostdev</span><span class="delimiter">'</span></span> <span class="attribute-name">managed</span>=<span class="string"><span class="delimiter">'</span><span class="content">yes</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 87</span>      <span class="tag">&lt;mac</span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">02:04:17:01:02:02</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 88</span>      <span class="tag">&lt;driver</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">kvm</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 89</span>      <span class="tag">&lt;source&gt;</span>
<span class="line-numbers"> 90</span>        <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x06</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x10</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 91</span>      <span class="tag">&lt;/source&gt;</span>
<span class="line-numbers"> 92</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">hostdev1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 93</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x07</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 94</span>    <span class="tag">&lt;/interface&gt;</span>
<span class="line-numbers"> 95</span>    <span class="tag">&lt;serial</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">tcp</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers"> 96</span>      <span class="tag">&lt;source</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">bind</span><span class="delimiter">'</span></span> <span class="attribute-name">host</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span> <span class="attribute-name">service</span>=<span class="string"><span class="delimiter">'</span><span class="content">8817</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 97</span>      <span class="tag">&lt;protocol</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">telnet</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 98</span>      <span class="tag">&lt;target</span> <span class="attribute-name">port</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers"> 99</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">serial0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">100</span>    <span class="tag">&lt;/serial&gt;</span>
<span class="line-numbers">101</span>    <span class="tag">&lt;console</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">tcp</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">102</span>      <span class="tag">&lt;source</span> <span class="attribute-name">mode</span>=<span class="string"><span class="delimiter">'</span><span class="content">bind</span><span class="delimiter">'</span></span> <span class="attribute-name">host</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span> <span class="attribute-name">service</span>=<span class="string"><span class="delimiter">'</span><span class="content">8817</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">103</span>      <span class="tag">&lt;protocol</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">telnet</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">104</span>      <span class="tag">&lt;target</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">serial</span><span class="delimiter">'</span></span> <span class="attribute-name">port</span>=<span class="string"><span class="delimiter">'</span><span class="content">0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">105</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">serial0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">106</span>    <span class="tag">&lt;/console&gt;</span>
<span class="line-numbers">107</span>    <span class="tag">&lt;input</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">tablet</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">usb</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">108</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">input0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">109</span>    <span class="tag">&lt;/input&gt;</span>
<span class="line-numbers">110</span>    <span class="tag">&lt;input</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">mouse</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ps2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">111</span>    <span class="tag">&lt;input</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">keyboard</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">ps2</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">112</span>    <span class="tag">&lt;graphics</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">vnc</span><span class="delimiter">'</span></span> <span class="attribute-name">port</span>=<span class="string"><span class="delimiter">'</span><span class="content">5901</span><span class="delimiter">'</span></span> <span class="attribute-name">autoport</span>=<span class="string"><span class="delimiter">'</span><span class="content">yes</span><span class="delimiter">'</span></span> <span class="attribute-name">listen</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">113</span>      <span class="tag">&lt;listen</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">address</span><span class="delimiter">'</span></span> <span class="attribute-name">address</span>=<span class="string"><span class="delimiter">'</span><span class="content">127.0.0.1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">114</span>    <span class="tag">&lt;/graphics&gt;</span>
<span class="line-numbers">115</span>    <span class="tag">&lt;sound</span> <span class="attribute-name">model</span>=<span class="string"><span class="delimiter">'</span><span class="content">ac97</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">116</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">sound0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">117</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x05</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">118</span>    <span class="tag">&lt;/sound&gt;</span>
<span class="line-numbers">119</span>    <span class="tag">&lt;video&gt;</span>
<span class="line-numbers">120</span>      <span class="tag">&lt;model</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">cirrus</span><span class="delimiter">'</span></span> <span class="attribute-name">vram</span>=<span class="string"><span class="delimiter">'</span><span class="content">9216</span><span class="delimiter">'</span></span> <span class="attribute-name">heads</span>=<span class="string"><span class="delimiter">'</span><span class="content">1</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">121</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">video0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">122</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x02</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">123</span>    <span class="tag">&lt;/video&gt;</span>
<span class="line-numbers">124</span>    <span class="tag">&lt;memballoon</span> <span class="attribute-name">model</span>=<span class="string"><span class="delimiter">'</span><span class="content">virtio</span><span class="delimiter">'</span></span><span class="tag">&gt;</span>
<span class="line-numbers">125</span>      <span class="tag">&lt;alias</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">'</span><span class="content">balloon0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">126</span>      <span class="tag">&lt;address</span> <span class="attribute-name">type</span>=<span class="string"><span class="delimiter">'</span><span class="content">pci</span><span class="delimiter">'</span></span> <span class="attribute-name">domain</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0000</span><span class="delimiter">'</span></span> <span class="attribute-name">bus</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x00</span><span class="delimiter">'</span></span> <span class="attribute-name">slot</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x08</span><span class="delimiter">'</span></span> <span class="attribute-name">function</span>=<span class="string"><span class="delimiter">'</span><span class="content">0x0</span><span class="delimiter">'</span></span><span class="tag">/&gt;</span>
<span class="line-numbers">127</span>    <span class="tag">&lt;/memballoon&gt;</span>
<span class="line-numbers">128</span>  <span class="tag">&lt;/devices&gt;</span>
<span class="line-numbers">129</span><span class="tag">&lt;/domain&gt;</span></code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="__code_virsh_capabilities_code_complete_output"><a class="anchor" href="#__code_virsh_capabilities_code_complete_output"></a><code>virsh capabilities</code> complete output</h2>
<div class="sectionbody">
<div class="listingblock">
<div class="content">
<pre>ping@trinity:~$ sudo virsh capabilities
&lt;capabilities&gt;

  &lt;host&gt;
    &lt;uuid&gt;36373931-3138-5553-4534-333739575353&lt;/uuid&gt;
    &lt;cpu&gt;
      &lt;arch&gt;x86_64&lt;/arch&gt;
      &lt;model&gt;SandyBridge&lt;/model&gt;
      &lt;vendor&gt;Intel&lt;/vendor&gt;
      &lt;topology sockets='1' cores='8' threads='1'/&gt;
      &lt;feature name='invtsc'/&gt;
      &lt;feature name='erms'/&gt;
      &lt;feature name='smep'/&gt;
      &lt;feature name='fsgsbase'/&gt;
      &lt;feature name='pdpe1gb'/&gt;
      &lt;feature name='rdrand'/&gt;
      &lt;feature name='f16c'/&gt;
      &lt;feature name='osxsave'/&gt;
      &lt;feature name='dca'/&gt;
      &lt;feature name='pcid'/&gt;
      &lt;feature name='pdcm'/&gt;
      &lt;feature name='xtpr'/&gt;
      &lt;feature name='tm2'/&gt;
      &lt;feature name='est'/&gt;
      &lt;feature name='smx'/&gt;
      &lt;feature name='vmx'/&gt;
      &lt;feature name='ds_cpl'/&gt;
      &lt;feature name='monitor'/&gt;
      &lt;feature name='dtes64'/&gt;
      &lt;feature name='pbe'/&gt;
      &lt;feature name='tm'/&gt;
      &lt;feature name='ht'/&gt;
      &lt;feature name='ss'/&gt;
      &lt;feature name='acpi'/&gt;
      &lt;feature name='ds'/&gt;
      &lt;feature name='vme'/&gt;
      &lt;pages unit='KiB' size='4'/&gt;
      &lt;pages unit='KiB' size='2048'/&gt;
    &lt;/cpu&gt;
    &lt;power_management&gt;
      &lt;suspend_disk/&gt;
      &lt;suspend_hybrid/&gt;
    &lt;/power_management&gt;
    &lt;migration_features&gt;
      &lt;live/&gt;
      &lt;uri_transports&gt;
        &lt;uri_transport&gt;tcp&lt;/uri_transport&gt;
      &lt;/uri_transports&gt;
    &lt;/migration_features&gt;
    &lt;topology&gt;
      &lt;cells num='4'&gt;
        &lt;cell id='0'&gt;
          &lt;memory unit='KiB'&gt;132067432&lt;/memory&gt;
          &lt;pages unit='KiB' size='4'&gt;33016858&lt;/pages&gt;
          &lt;pages unit='KiB' size='2048'&gt;0&lt;/pages&gt;
          &lt;distances&gt;
            &lt;sibling id='0' value='10'/&gt;
            &lt;sibling id='1' value='21'/&gt;
            &lt;sibling id='2' value='21'/&gt;
            &lt;sibling id='3' value='21'/&gt;
          &lt;/distances&gt;
          &lt;cpus num='8'&gt;
            &lt;cpu id='0' socket_id='0' core_id='0' siblings='0'/&gt;
            &lt;cpu id='1' socket_id='0' core_id='1' siblings='1'/&gt;
            &lt;cpu id='2' socket_id='0' core_id='2' siblings='2'/&gt;
            &lt;cpu id='3' socket_id='0' core_id='3' siblings='3'/&gt;
            &lt;cpu id='4' socket_id='0' core_id='4' siblings='4'/&gt;
            &lt;cpu id='5' socket_id='0' core_id='5' siblings='5'/&gt;
            &lt;cpu id='6' socket_id='0' core_id='6' siblings='6'/&gt;
            &lt;cpu id='7' socket_id='0' core_id='7' siblings='7'/&gt;
          &lt;/cpus&gt;
        &lt;/cell&gt;
        &lt;cell id='1'&gt;
          &lt;memory unit='KiB'&gt;132116676&lt;/memory&gt;
          &lt;pages unit='KiB' size='4'&gt;33029169&lt;/pages&gt;
          &lt;pages unit='KiB' size='2048'&gt;0&lt;/pages&gt;
          &lt;distances&gt;
            &lt;sibling id='0' value='21'/&gt;
            &lt;sibling id='1' value='10'/&gt;
            &lt;sibling id='2' value='21'/&gt;
            &lt;sibling id='3' value='21'/&gt;
          &lt;/distances&gt;
          &lt;cpus num='8'&gt;
            &lt;cpu id='8' socket_id='1' core_id='0' siblings='8'/&gt;
            &lt;cpu id='9' socket_id='1' core_id='1' siblings='9'/&gt;
            &lt;cpu id='10' socket_id='1' core_id='2' siblings='10'/&gt;
            &lt;cpu id='11' socket_id='1' core_id='3' siblings='11'/&gt;
            &lt;cpu id='12' socket_id='1' core_id='4' siblings='12'/&gt;
            &lt;cpu id='13' socket_id='1' core_id='5' siblings='13'/&gt;
            &lt;cpu id='14' socket_id='1' core_id='6' siblings='14'/&gt;
            &lt;cpu id='15' socket_id='1' core_id='7' siblings='15'/&gt;
          &lt;/cpus&gt;
        &lt;/cell&gt;
        &lt;cell id='2'&gt;
          &lt;memory unit='KiB'&gt;132116676&lt;/memory&gt;
          &lt;pages unit='KiB' size='4'&gt;33029169&lt;/pages&gt;
          &lt;pages unit='KiB' size='2048'&gt;0&lt;/pages&gt;
          &lt;distances&gt;
            &lt;sibling id='0' value='21'/&gt;
            &lt;sibling id='1' value='21'/&gt;
            &lt;sibling id='2' value='10'/&gt;
            &lt;sibling id='3' value='21'/&gt;
          &lt;/distances&gt;
          &lt;cpus num='8'&gt;
            &lt;cpu id='16' socket_id='2' core_id='0' siblings='16'/&gt;
            &lt;cpu id='17' socket_id='2' core_id='1' siblings='17'/&gt;
            &lt;cpu id='18' socket_id='2' core_id='2' siblings='18'/&gt;
            &lt;cpu id='19' socket_id='2' core_id='3' siblings='19'/&gt;
            &lt;cpu id='20' socket_id='2' core_id='4' siblings='20'/&gt;
            &lt;cpu id='21' socket_id='2' core_id='5' siblings='21'/&gt;
            &lt;cpu id='22' socket_id='2' core_id='6' siblings='22'/&gt;
            &lt;cpu id='23' socket_id='2' core_id='7' siblings='23'/&gt;
          &lt;/cpus&gt;
        &lt;/cell&gt;
        &lt;cell id='3'&gt;
          &lt;memory unit='KiB'&gt;132116616&lt;/memory&gt;
          &lt;pages unit='KiB' size='4'&gt;33029154&lt;/pages&gt;
          &lt;pages unit='KiB' size='2048'&gt;0&lt;/pages&gt;
          &lt;distances&gt;
            &lt;sibling id='0' value='21'/&gt;
            &lt;sibling id='1' value='21'/&gt;
            &lt;sibling id='2' value='21'/&gt;
            &lt;sibling id='3' value='10'/&gt;
          &lt;/distances&gt;
          &lt;cpus num='8'&gt;
            &lt;cpu id='24' socket_id='3' core_id='0' siblings='24'/&gt;
            &lt;cpu id='25' socket_id='3' core_id='1' siblings='25'/&gt;
            &lt;cpu id='26' socket_id='3' core_id='2' siblings='26'/&gt;
            &lt;cpu id='27' socket_id='3' core_id='3' siblings='27'/&gt;
            &lt;cpu id='28' socket_id='3' core_id='4' siblings='28'/&gt;
            &lt;cpu id='29' socket_id='3' core_id='5' siblings='29'/&gt;
            &lt;cpu id='30' socket_id='3' core_id='6' siblings='30'/&gt;
            &lt;cpu id='31' socket_id='3' core_id='7' siblings='31'/&gt;
          &lt;/cpus&gt;
        &lt;/cell&gt;
      &lt;/cells&gt;
    &lt;/topology&gt;
    &lt;secmodel&gt;
      &lt;model&gt;none&lt;/model&gt;
      &lt;doi&gt;0&lt;/doi&gt;
    &lt;/secmodel&gt;
    &lt;secmodel&gt;
      &lt;model&gt;dac&lt;/model&gt;
      &lt;doi&gt;0&lt;/doi&gt;
      &lt;baselabel type='kvm'&gt;+0:+0&lt;/baselabel&gt;
      &lt;baselabel type='qemu'&gt;+0:+0&lt;/baselabel&gt;
    &lt;/secmodel&gt;
  &lt;/host&gt;

  &lt;guest&gt;
    &lt;os_type&gt;hvm&lt;/os_type&gt;
    &lt;arch name='i686'&gt;
      &lt;wordsize&gt;32&lt;/wordsize&gt;
      &lt;emulator&gt;/usr/bin/qemu-system-i386&lt;/emulator&gt;
      &lt;machine canonical='pc-i440fx-trusty' maxCpus='255'&gt;pc&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.12&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-1.3&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.6&lt;/machine&gt;
      &lt;machine canonical='pc-1.0-qemu-kvm' maxCpus='255'&gt;pc-1.0-precise&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.5&lt;/machine&gt;
      &lt;machine maxCpus='1'&gt;xenpv&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.6&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.7&lt;/machine&gt;
      &lt;machine canonical='pc-i440fx-1.5-qemu-kvm' maxCpus='255'&gt;pc-i440fx-1.5-saucy&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.11&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.10&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-1.2&lt;/machine&gt;
      &lt;machine maxCpus='1'&gt;isapc&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.4&lt;/machine&gt;
      &lt;machine maxCpus='128'&gt;xenfv&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.15&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.14&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.5&lt;/machine&gt;
      &lt;machine canonical='pc-q35-2.0' maxCpus='255'&gt;q35&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.4&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-1.1&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.7&lt;/machine&gt;
      &lt;machine canonical='pc-1.0' maxCpus='255'&gt;pc-1.0-qemu-kvm&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-2.0&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.13&lt;/machine&gt;
      &lt;domain type='qemu'&gt;
      &lt;/domain&gt;
      &lt;domain type='kvm'&gt;
        &lt;emulator&gt;/usr/bin/kvm&lt;/emulator&gt;
        &lt;machine canonical='pc-i440fx-trusty' maxCpus='255'&gt;pc&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.3&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.12&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.6&lt;/machine&gt;
        &lt;machine canonical='pc-1.0-qemu-kvm' maxCpus='255'&gt;pc-1.0-precise&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.5&lt;/machine&gt;
        &lt;machine maxCpus='1'&gt;xenpv&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.6&lt;/machine&gt;
        &lt;machine canonical='pc-i440fx-1.5-qemu-kvm' maxCpus='255'&gt;pc-i440fx-1.5-saucy&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.7&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.11&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.2&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.10&lt;/machine&gt;
        &lt;machine maxCpus='1'&gt;isapc&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.4&lt;/machine&gt;
        &lt;machine maxCpus='128'&gt;xenfv&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.15&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.14&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.5&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.4&lt;/machine&gt;
        &lt;machine canonical='pc-q35-2.0' maxCpus='255'&gt;q35&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.1&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.7&lt;/machine&gt;
        &lt;machine canonical='pc-1.0' maxCpus='255'&gt;pc-1.0-qemu-kvm&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-2.0&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.13&lt;/machine&gt;
      &lt;/domain&gt;
    &lt;/arch&gt;
    &lt;features&gt;
      &lt;cpuselection/&gt;
      &lt;deviceboot/&gt;
      &lt;disksnapshot default='on' toggle='no'/&gt;
      &lt;acpi default='on' toggle='yes'/&gt;
      &lt;apic default='on' toggle='no'/&gt;
      &lt;pae/&gt;
      &lt;nonpae/&gt;
    &lt;/features&gt;
  &lt;/guest&gt;

  &lt;guest&gt;
    &lt;os_type&gt;hvm&lt;/os_type&gt;
    &lt;arch name='x86_64'&gt;
      &lt;wordsize&gt;64&lt;/wordsize&gt;
      &lt;emulator&gt;/usr/bin/qemu-system-x86_64&lt;/emulator&gt;
      &lt;machine canonical='pc-i440fx-trusty' maxCpus='255'&gt;pc&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-1.3&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.12&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.6&lt;/machine&gt;
      &lt;machine canonical='pc-1.0-qemu-kvm' maxCpus='255'&gt;pc-1.0-precise&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.5&lt;/machine&gt;
      &lt;machine maxCpus='1'&gt;xenpv&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.6&lt;/machine&gt;
      &lt;machine canonical='pc-i440fx-1.5-qemu-kvm' maxCpus='255'&gt;pc-i440fx-1.5-saucy&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.7&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.11&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-1.2&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.10&lt;/machine&gt;
      &lt;machine maxCpus='1'&gt;isapc&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.4&lt;/machine&gt;
      &lt;machine maxCpus='128'&gt;xenfv&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.15&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.14&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.5&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-1.4&lt;/machine&gt;
      &lt;machine canonical='pc-q35-2.0' maxCpus='255'&gt;q35&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-1.1&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-q35-1.7&lt;/machine&gt;
      &lt;machine canonical='pc-1.0' maxCpus='255'&gt;pc-1.0-qemu-kvm&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-i440fx-2.0&lt;/machine&gt;
      &lt;machine maxCpus='255'&gt;pc-0.13&lt;/machine&gt;
      &lt;domain type='qemu'&gt;
      &lt;/domain&gt;
      &lt;domain type='kvm'&gt;
        &lt;emulator&gt;/usr/bin/kvm&lt;/emulator&gt;
        &lt;machine canonical='pc-i440fx-trusty' maxCpus='255'&gt;pc&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.3&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.12&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.6&lt;/machine&gt;
        &lt;machine canonical='pc-1.0-qemu-kvm' maxCpus='255'&gt;pc-1.0-precise&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.5&lt;/machine&gt;
        &lt;machine maxCpus='1'&gt;xenpv&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.6&lt;/machine&gt;
        &lt;machine canonical='pc-i440fx-1.5-qemu-kvm' maxCpus='255'&gt;pc-i440fx-1.5-saucy&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.7&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.11&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.2&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.10&lt;/machine&gt;
        &lt;machine maxCpus='1'&gt;isapc&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.4&lt;/machine&gt;
        &lt;machine maxCpus='128'&gt;xenfv&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.15&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.14&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.5&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-1.4&lt;/machine&gt;
        &lt;machine canonical='pc-q35-2.0' maxCpus='255'&gt;q35&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-1.1&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-q35-1.7&lt;/machine&gt;
        &lt;machine canonical='pc-1.0' maxCpus='255'&gt;pc-1.0-qemu-kvm&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-i440fx-2.0&lt;/machine&gt;
        &lt;machine maxCpus='255'&gt;pc-0.13&lt;/machine&gt;
      &lt;/domain&gt;
    &lt;/arch&gt;
    &lt;features&gt;
      &lt;cpuselection/&gt;
      &lt;deviceboot/&gt;
      &lt;disksnapshot default='on' toggle='no'/&gt;
      &lt;acpi default='on' toggle='yes'/&gt;
      &lt;apic default='on' toggle='no'/&gt;
    &lt;/features&gt;
  &lt;/guest&gt;

&lt;/capabilities&gt;</pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_ixgbe_driver_issue"><a class="anchor" href="#_ixgbe_driver_issue"></a>ixgbe driver issue</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The orignal IXGBE driver coming with ubuntu has issue on multicast - the VMX
"Getting Started Guide" does mention this:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Multicast promiscuous mode for Virtual Functions is needed to receive control
traffic that comes with broadcast MAC addresses</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>So it looks like an issue of multicast support specifically on VF. A quick test
here illustates the issue.</p>
</div>
<div class="paragraph">
<p>To test this I manually setup a VMX, with the original IXGBE driver coming with
ubuntu. the setup is as simple as just a HP server with VMX installed connected
to another L3 switch(QFX), showing below:</p>
</div>
<div class="literalblock">
<div class="title">ixgbe multicast issue test diagram</div>
<div class="content">
<pre>   HP server
.............
.  +------+ .                    +------+
.  |      | .                    |      |
.  |VMX   | .TX RX               |QFX   |
.  |      | . |  |               |      |
.  +------+ . |  |               |      |
.           . |  |               |      |
.10.10.10.1 . |  X               |      |
.  NIC/VF   . |  |               |      |
.  +------+ . v  +-&lt;-----------  |      |
.  +---+--+ . |                  |      |
.      |    . +----&gt;-----------  |      |
.............                    +--+---+
       |                            | 10.10.10.2
       +----------------------------+</pre>
</div>
</div>
<div class="paragraph">
<p>configuration on both end are simple: an IP interface with OSPF enabled on it.</p>
</div>
<div class="literalblock">
<div class="title">QFX configuration</div>
<div class="content">
<pre>set groups test-ixgbe interfaces xe-0/0/18 unit 0 family inet address 10.10.10.2/24
set groups test-ixgbe routing-instances test instance-type virtual-router
set groups test-ixgbe routing-instances test interface xe-0/0/18.0
set groups test-ixgbe routing-instances test protocols ospf area 0.0.0.0 interface xe-0/0/18.0</pre>
</div>
</div>
<div class="literalblock">
<div class="title">VMX configuration</div>
<div class="content">
<pre>root# show | compare
[edit]
+  protocols {
+      ospf {
+          area 0.0.0.0 {
+              interface ge-0/0/0.0;
+          }
+      }
+  }</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>[edit]
root# run show ospf interface
Interface           State   Area            DR ID           BDR ID          Nbrs
ge-0/0/0.0          DR      0.0.0.0         10.10.10.1      0.0.0.0            0</pre>
</div>
</div>
<div class="paragraph">
<div class="title">the issue</div>
<p>the issue is that ping works, but OSPF adjacency does not come up:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vmx# run ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2): 56 data bytes
64 bytes from 10.10.10.2: icmp_seq=0 ttl=64 time=13.889 ms
^C
--- 10.10.10.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 13.889/13.889/13.889/0.000 ms</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>root# root# run show ospf neighbor
root#</pre>
</div>
</div>
<div class="paragraph">
<div class="title">packet capture</div>
<p>packet capture on VMX host server shows physical port (p2p1) is able to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>receive ospf packets from peer device</p>
</li>
<li>
<p>deliver OSPF packets out once receiving from VMX</p>
<div class="literalblock">
<div class="content">
<pre>ping@matrix:/home/vAVPN/images$ sudo tcpdump -ni p2p1
tcpdump: WARNING: p2p1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on p2p1, link-type EN10MB (Ethernet), capture size 65535 bytes
12:18:12.963197 IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
12:18:17.477019 IP 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60
12:18:21.483668 IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
12:18:21.989157 LLDP, length 296: sonata
^C</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>but packet capture from VMX shows it doesn&#8217;t receive anything:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>root# run monitor traffic interface ge-0/0/0 size 2000
verbose output suppressed, use &lt;detail&gt; or &lt;extensive&gt; for full protocol decode
Address resolution is ON. Use &lt;no-resolve&gt; to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ge-0/0/0, capture size 2000 bytes</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Reverse lookup for 224.0.0.5 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use &lt;no-resolve&gt; to avoid reverse lookups on IP addresses.</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>20:17:20.318417 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:17:29.588411 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:17:37.958421 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:17:47.058547 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:17:54.598639 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:18:03.078845 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:18:12.408862 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
20:18:20.929166 Out IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56</pre>
</div>
</div>
<div class="paragraph">
<p>meanwhile capture in QFX shows it is receiving and sending packets without a
problem:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>labroot@sonata# run monitor traffic interface xe-0/0/18
verbose output suppressed, use &lt;detail&gt; or &lt;extensive&gt; for full protocol decode
Address resolution is ON. Use &lt;no-resolve&gt; to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on xe-0/0/18, capture size 96 bytes</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>Reverse lookup for 224.0.0.5 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use &lt;no-resolve&gt; to avoid reverse lookups on IP addresses.</pre>
</div>
</div>
<div class="literalblock">
<div class="content">
<pre>16:00:25.375850  In IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
16:00:28.511860 Out IP truncated-ip - 20 bytes missing! 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60
16:00:30.867355 Out LLDP, name sonata, length 60
        [|LLDP]
16:00:35.119801  In IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
16:00:37.549066 Out IP truncated-ip - 20 bytes missing! 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60
16:00:42.633445  In IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
16:00:47.589120 Out IP truncated-ip - 20 bytes missing! 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60
16:00:52.077874  In IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
16:00:56.129699 Out IP truncated-ip - 20 bytes missing! 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60
16:00:58.003400 Out LLDP, name sonata, length 60
        [|LLDP]
16:01:01.841421  In IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
16:01:05.589241 Out IP truncated-ip - 20 bytes missing! 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60
16:01:11.136212  In IP 10.10.10.1 &gt; 224.0.0.5: OSPFv2, Hello, length 56
16:01:14.940515 Out IP truncated-ip - 20 bytes missing! 10.10.10.2 &gt; 224.0.0.5: OSPFv2, Hello, length 60</pre>
</div>
</div>
<div class="paragraph">
<p>So it just looks like OSPF packet coming in the server NIC does not get handed
over to VMX. <code>ethtool</code> with <code>-S</code> option will print counters for all VFs,
here is the capture when the issue is ongoing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping@matrix:~$ ethtool -S p2p1  | grep -iE "rx_packets|tx_packets|VF 0"
     rx_packets: 33685
     tx_packets: 8
     VF 0 Rx Packets: 315
     VF 0 Rx Bytes: 19546
     VF 0 Tx Packets: 20407
     VF 0 Tx Bytes: 1821654
     VF 0 MC Packets: 0</pre>
</div>
</div>
<div class="paragraph">
<p>It shows VF 0 didn&#8217;t receive any packet. Given that physical NIC p2p1 (PF)
actually received the packet, we can conclude all multicast packets were
dropped at VF 0.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<i class="fa icon-note" title="Note"></i>
</td>
<td class="content">
<div class="paragraph">
<p>For physical NIC the multicast capability can be enabled by this command:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ifconfig p3p1 up promisc allmulti mtu 2000</pre>
</div>
</div>
<div class="paragraph">
<p>unfortunately this does not apply to VFs. The current solution is Juniper
modify the IXGBE code to fix this. So recompiling IXGBE driver from Juniper
provided source code will solve this issue.</p>
</div>
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_end"><a class="anchor" href="#_end"></a>end</h2>
<div class="sectionbody">

</div>
</div>
</div>
<div id="footnotes">
<hr>
<div class="footnote" id="_footnote_1">
<a href="#_footnoteref_1">1</a>. interface showing in host, and peering with guest VM interface, e.g. the <code>vcp_ext-vmx1</code> interface is "peer interface" of fxp0 in VMX
</div>
<div class="footnote" id="_footnote_2">
<a href="#_footnoteref_2">2</a>. the IP address specified here still needs to be configured manually from inside of the vRE guest VM, unless dhcp client is implemented from guest VM to request for an IP from the DHCP server running in host OS, this may be implemented in the future releases
</div>
<div class="footnote" id="_footnote_3">
<a href="#_footnoteref_3">3</a>. in the same host there can be some other VMs that may not be spinned up by libvirt, in that case those VM won&#8217;t be under control of libvirt
</div>
<div class="footnote" id="_footnote_4">
<a href="#_footnoteref_4">4</a>. the dns service here is not of our concern at this time, knowning we also have these "extra" services enabled does not harm anyway
</div>
<div class="footnote" id="_footnote_5">
<a href="#_footnoteref_5">5</a>. or, it can be "logical" cpu core when hyperthreading is enabled
</div>
<div class="footnote" id="_footnote_6">
<a href="#_footnoteref_6">6</a>. from SR-IOV spec
</div>
</div>
<div id="footer">
<div id="footer-text">
</div>
</div>
</body>
</html>