Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dockerfiles #771

Merged
merged 41 commits into from
Feb 21, 2021
Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
53386ef
added gettext dependency for qemu build
eddygta17 Jun 13, 2020
8e19b46
Update Gemmini.rst
amsharifian Oct 7, 2020
4253382
switched dockerfile base to ubuntu and fixed bugs. created entrypoint…
schwarz-em Nov 30, 2020
8141c71
entrypoint.sh sets environment variables correctly. Dockerfile has al…
schwarz-em Dec 1, 2020
77459a3
working version of dockerfile and entrypoint.sh
schwarz-em Dec 3, 2020
cfca224
removed log files for real this time?
schwarz-em Dec 3, 2020
fe1f7fe
changed permissions of entrypoint.sh
schwarz-em Dec 6, 2020
7439191
removed some redundancies from dockerfile
schwarz-em Dec 8, 2020
3957282
Cleaned up dockerfile
Dec 21, 2020
ef84927
Removed environment variables from dockerfile
Dec 23, 2020
db7fafb
switched dockerfile base to ubuntu and fixed bugs. created entrypoint…
schwarz-em Nov 30, 2020
f5ae2e0
entrypoint.sh sets environment variables correctly. Dockerfile has al…
schwarz-em Dec 1, 2020
f25aa8d
working version of dockerfile and entrypoint.sh
schwarz-em Dec 3, 2020
3ba19bb
changed permissions of entrypoint.sh
schwarz-em Dec 6, 2020
8300d9c
removed log files
Jan 12, 2021
b528058
part 2 of removing log files
Jan 12, 2021
9b84682
Fixed comments and simplified entrypoint
Jan 14, 2021
49a76a6
Updated ci dockerfile. Moved dockerfiles to outer directory. Updated …
Jan 21, 2021
a46c4e6
moved Dockerfiles
Jan 21, 2021
fa2da01
Changed config.yml to use new docker image
Jan 21, 2021
925bace
fixed merge conflict
Jan 21, 2021
58665ea
Added hardcoded environment variables to CI-Dockerfile and updated co…
Jan 22, 2021
500d2b8
Build both CI and main images in one dockerfile. Updated README.md
Jan 28, 2021
e13266b
Install python
Jan 28, 2021
5e90372
Updated image for to 1.0.2 for config.yml. Fixed formatting in Docker…
Jan 28, 2021
e95e033
Updated dockerfile comments and updated dockerfile readme. Removed ci…
Feb 4, 2021
d2b3dca
Readme updated for main dockerfile
Feb 4, 2021
7eb8f39
Uses latest chipyard ci image
Feb 5, 2021
34e56d5
Moved CMD command
Feb 6, 2021
1f8b597
Uses latest version of ci-image
Feb 6, 2021
84cfc73
Removed default argument
Feb 8, 2021
75bc39a
Uses latest ci image
Feb 8, 2021
323394f
Increased timeout limit
Feb 8, 2021
d7be658
Added information about docker image
Feb 14, 2021
788b22a
Added more details on docker image
Feb 15, 2021
0f3a703
Increased timeout to 20m
Feb 15, 2021
f06afa2
Increased timeout to 20m for chipyard-hetero tests
Feb 15, 2021
18aa600
Fix Chipyard spelling in docs [ci skip]
abejgonzalez Feb 15, 2021
d647534
Added more details on which tools are initialized in the docker image
Feb 19, 2021
9c650de
Removed apt-get upgrade and edited for image size
Feb 19, 2021
b12fa04
Updated docker image
Feb 19, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ parameters:
executors:
main-env:
docker:
- image: ucbbar/chipyard-image:1.0.1
- image: ucbbar/chipyard-ci-image:84cfc73
environment:
JVM_OPTS: -Xmx3200m # Customize the JVM maximum heap limit

Expand Down Expand Up @@ -112,7 +112,7 @@ commands:
default: "run-tests.sh"
timeout:
type: string
default: "10m"
default: "20m"
steps:
- setup-tools:
tools-version: "<< parameters.tools-version >>"
Expand Down Expand Up @@ -231,7 +231,7 @@ jobs:
- run-tests:
group-key: "group-cores"
project-key: "chipyard-hetero"
timeout: "15m"
timeout: "20m"
chipyard-boom-run-tests:
executor: main-env
steps:
Expand Down Expand Up @@ -520,4 +520,3 @@ workflows:
- prepare-chipyard-fpga:
requires:
- install-riscv-toolchain

216 changes: 0 additions & 216 deletions .circleci/images/Dockerfile

This file was deleted.

18 changes: 0 additions & 18 deletions .circleci/images/README.md

This file was deleted.

68 changes: 68 additions & 0 deletions dockerfiles/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
### This is a full chipyard setup

# BUILD BASE FOR CI

FROM ubuntu:18.04 as base
ARG CHIPYARD_HASH

MAINTAINER https://groups.google.com/forum/#!forum/chipyard

# Install dependencies for ubuntu-req.sh
RUN apt-get update && \
apt-get upgrade -y && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(sorry for chiming in so late)

Usually, you don't actually want to run apt-get upgrade on a Docker container. From this page:

Avoid RUN apt-get upgrade and dist-upgrade, as many of the “essential” packages from the parent images cannot upgrade inside an unprivileged container. If a package contained in the parent image is out-of-date, contact its maintainers. If you know there is a particular package, foo, that needs to be updated, use apt-get install -y foo to update automatically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tip. I was able to remove apt-get upgrade without any problems.

apt-get install -y \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you add the --no-install-recommends flag to apt-get install and conclude the command with rm -rf /var/lib/apt/lists/*, then you can help to minimize the finished container size.

curl \
git \
sudo

# Stopping docker keyboard-config from disrupting ubuntu-req.sh
RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y keyboard-configuration && \
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y console-setup

# Adds a new user called riscvuser
RUN groupadd --gid 3434 riscvuser \
&& useradd --uid 3434 --gid riscvuser --shell /bin/bash --create-home riscvuser \
&& echo 'riscvuser ALL=NOPASSWD: ALL' >> /etc/sudoers.d/50-riscvuser \
&& echo 'Defaults env_keep += "DEBIAN_FRONTEND"' >> /etc/sudoers.d/env_keep

WORKDIR /home/riscvuser
USER riscvuser

# Update PATH for RISCV toolchain (note: hardcoded for CircleCI)
ENV RISCV="/home/riscvuser/riscv-tools-install"
ENV LD_LIBRARY_PATH="$RISCV/lib"
ENV PATH="$RISCV/bin:$PATH"

# Install Chipyard and run ubuntu-req.sh to install necessary dependencies
RUN git clone https://github.com/ucb-bar/chipyard.git && \
cd chipyard && \
git checkout $CHIPYARD_HASH && \
./scripts/ubuntu-req.sh 1>/dev/null
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this runs apt-get ... internally, you might want to add rm -rf /var/lib/apt/lists/* at the conclusion. In general though, this page mentions that running apt-get update from two separate steps (since you run it from within this script, and also from the very first RUN command) can cause weird issues, since build caching can mean that different apt install commands are working with different apt source lists.



# BUILD IMAGE WITH TOOLCHAINS

# Use above build as base
FROM base as base-with-tools

# Init submodules
RUN cd chipyard && \
export MAKEFLAGS=-"j $(nproc)" && \
./scripts/init-submodules-no-riscv-tools.sh 1>/dev/null

# Install riscv-tools
RUN cd chipyard && \
export MAKEFLAGS=-"j $(nproc)" && \
./scripts/build-toolchains.sh riscv-tools 1>/dev/null

# Install esp-tools
RUN cd chipyard && \
export MAKEFLAGS=-"j $(nproc)" && \
./scripts/build-toolchains.sh esp-tools 1>/dev/null

# Run script to set environment variables on entry
ENTRYPOINT ["chipyard/scripts/entrypoint.sh"]

# END IMAGE CUSTOMIZATIONS

CMD ["/bin/sh"]
22 changes: 22 additions & 0 deletions dockerfiles/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
General
-------
This DockerFile contains the necessary steps to build a Docker container that can run
abejgonzalez marked this conversation as resolved.
Show resolved Hide resolved
projects with riscv-tools, chisel3, firrtl, and verilator. When run up to the base stage, it installs the necessary
apt-get packages and sets the environment variables needed for CircleCI. When run up to the base-with-tools stage, it initializes and installs the necessary toolchains for running simulations and testing projects.

Build and Deploy the Container
------------------------------

sudo docker build --target base . # to build the image for the CI
sudo docker build --target base --build-arg CHIPYARD_HASH=<COMMIT_HASH> . # to build the image for the CI from a specific chipyard commit
sudo docker build --target base-with-tools . # to build the full image
sudo docker tag <IMAGE_ID> <PATH_NAME>:tag . # to tag the image after the build (ex. 0.0.3)
sudo docker login # login into the account to push to
sudo docker push <PATH_NAME>:tag # to push to repo with tag
sudo docker run -it <IMAGE_ID> bash # to run an interactive version of the container

Path Names
----------
Older docker images (when this Dockerfile was in `riscv-boom/riscv-boom`) can be found in the <PATH_NAME> `riscvboom/riscvboom-images`.
Current up-to-date images are located in <PATH_NAME> `ucbbar/chipyard-image`. NOTE: Less recent images in this path may not have toolchains initialized
Current up-to-date CI images are located in <PATH_NAME> `ucbbar/chipyard-ci-image`.
16 changes: 16 additions & 0 deletions docs/Chipyard-Basics/Initial-Repo-Setup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,22 @@ You can put this in your ``.bashrc`` or equivalent environment setup file to get

These variables need to be set for the ``make`` system to work properly.

Pre-built Docker Image
-------------------------------------------

An alternative to setting up the Chipyard repository locally is to pull the pre-built Docker image from Docker Hub. The image comes with all dependencies installed, chipyard cloned, and toolchains initialized. Each image comes with a tag that corresponds to the version of Chipyard cloned/set-up in that image. Not including a tag during the pull will pull the image with the latest version of chipyard.
abejgonzalez marked this conversation as resolved.
Show resolved Hide resolved
First, pull the Docker image. Run:

.. code-block:: shell

sudo docker pull ucbbar/chipyard-image:<TAG>

To run the Docker container in an interactive shell, run:

.. code-block:: shell

sudo docker run -it ucbbar/chipyard-image bash

What's Next?
-------------------------------------------

Expand Down
1 change: 1 addition & 0 deletions scripts/centos-req.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ sudo yum install -y centos-release-scl
sudo yum install -y devtoolset-8-make
# install DTC
sudo yum install -y dtc
sudo yum install -y python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few lines above there is a python36, python36-pip and python36-devel installation. Is this python installation for an earlier or later version?

Copy link
Contributor Author

@schwarz-em schwarz-em Feb 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would ideally be for a earlier version. I added this because there was a bug I ran into with ubuntu-req.sh where one of the tests was looking for python2 instead of python3. I don't know a way to test on centos, so I'm not entirely sure if the bug would still occur there, so I added it just in case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which test was looking for python2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was both of the tracegen tests.


# install verilator
git clone http://git.veripool.org/git/verilator
Expand Down
8 changes: 8 additions & 0 deletions scripts/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash

# used with the dockerfile to set up enviroment variables by running env.sh
# adapted from https://stackoverflow.com/questions/55921914/how-to-source-a-script-with-environment-variables-in-a-docker-build-process

. /home/riscvuser/chipyard/env.sh

exec "$@"
Loading