Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: rewrite build tool in Go #2401

Closed
bassosimone opened this issue Jan 25, 2023 · 1 comment
Closed

cli: rewrite build tool in Go #2401

bassosimone opened this issue Jan 25, 2023 · 1 comment
Assignees
@bassosimone
Labels
enhancement improving existing code or new feature interrupt ooni/probe-cli ooni/probe-engine priority/high refactoring techdebt This issue describes technical debt

Comments

@bassosimone
Copy link
Contributor

I am not satisfied by how we ensure that we're using the correct flags in ooni/probe-cli#1024. It turns out properly testing shell scripts and Makefile is painful. If the build tool was written in Go, instead, we could ~easily write unit and integration tests to make sure we're using the expected procedure for building. This would allow us to move fast(er).
@bassosimone bassosimone added enhancement improving existing code or new feature priority/high ooni/probe-cli refactoring ooni/probe-engine techdebt This issue describes technical debt labels Jan 25, 2023
@bassosimone bassosimone self-assigned this Jan 25, 2023
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
refactor: factor common code to check whether file exists
564224f 
Part of ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
refactor: factor common code to check whether file exists (#1045)
9270b52 
Part of ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat(shellx): support incremental command construction (#1046)
7008845 
Recent changes in #1042 made `shellx` better in terms of pretending we're running commands in tests. However, the work done there is not enough for ooni/probe#2401. It turns out I also need a mechanism to incrementally construct commands. So, while keeping unchanged the external `shellx` API used by existing packages, let's try to use a better underlying API that meets these new needs.

While there, re-read Go 1.19 release notes and realize that I want to keep using `x/sys/execabs` everywhere. See the informative comment in the diff below explaining why I want to do that.
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat(runtimex): add the TryN functions
b880a8d 
These functions are syntactic sugar to wrote runtimex.PanicOnError
code more compactly. This change is made possible by generics.

Part of ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat(runtimex): add the TryN functions (#1047)
c4fc1fc 
These functions are syntactic sugar to wrote runtimex.PanicOnError
code more compactly. This change is made possible by generics.

Part of ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat(must): convenience package with "must" functions
e7d709a 
A "must" function is a function that either succeeds or panics. We need
this kind of functions when writing programs for which it's acceptable to
panic in case there is an error (i.e., everything but ooniprobe and the
oohelperd daemon).

Part of ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat(must): convenience package with "must" functions (#1048)
8202af5 
A "must" function is a function that either succeeds or panics. We need
this kind of functions when writing programs for which it's acceptable to
panic in case there is an error (i.e., everything but ooniprobe and the
oohelperd daemon).

Part of ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: rewrite darwin build rules in Go (#1049)
1426f1e 
This diff rewrites Darwin build rules in Go. See ooni/probe#2401.
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: rewrite windows build rules in Go
fd6c60d 
This diff rewrites Windows build rules in Go.

See ooni/probe#2401.
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: rewrite windows build rules in Go (#1050)
2acefa3 
This diff rewrites Windows build rules in Go.

See ooni/probe#2401.
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: rewrite generic build rules in Go
801faed 
This diff rewrites the generic build rules in Go.

See ooni/probe#2401.
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: rewrite generic build rules in Go (#1051)
35faf23 
This diff rewrites the generic build rules in Go.

See ooni/probe#2401.
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: rewrite linux build rules in Go (#1044)
7ab2370 
This diff rewrites Linux build rules in Go.

See ooni/probe#2401.
@bassosimone bassosimone mentioned this issue Jan 25, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 25, 2023
@bassosimone
feat: mechanism compile and embed tor (#1052)
95d23c7 
This diff introduces a mechanism to compile and embed tor. We plan on using this functionality on Android and iOS but we have not finished working on updating the build yet. In the meanwhile, here's an interim result. We're committing the code to run unit and integration testing of this new functionality on linux/amd64.

The reference issue is ooni/probe#2365. 

The Go based build was sketched out in ooni/probe#2401.
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: rewrite the android build in Go
281217d 
See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: rewrite the android build in Go (#1053)
da393c0 
See ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: convert ios build to use Go
4bee5f1 
See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: convert ios build to use Go (#1054)
e55d3dc 
See ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: build for android using Go
d404652 
See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: build for Android using Go (#1055)
0e52d3c 
While there, cleanup unneeded scripts now that the build procedure is mostly written in Go.

See ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): unify C deps with other deps
88190d4 
It was a bit of annoying two have two structs to represent
build dependencies when just one of them is good enough.

Part of ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): unify C deps with other deps (#1057)
2b2d9e0 
It was a bit of annoying two have two structs to represent
build dependencies when just one of them is good enough.

Part of ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): merge cdeps and android build envs
522fb7b 
At the end of the day, there's just a unique kind of build env
that makes sense for both cdeps and android.

While there, write better code for merging global and local
environment and make sure we have unit tests for that.

See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): merge cdeps and android build envs (#1058)
c966cd6 
At the end of the day, there's just a unique kind of build env
that makes sense for both cdeps and android.

While there, write better code for merging global and local
environment and make sure we have unit tests for that.

See ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): use naming reminding to bash variables
fbeff3c 
I know this is a bit weird in the Go convention but using
this naming helps me to read the code in this case.

See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): use naming reminding of bash variables (#1059)
a39a9d2 
refactor(buildtool): use naming reminding to bash variables

I know this is a bit weird in the Go convention but using
this naming helps me to read the code in this case.

See ooni/probe#2401
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): improve how we set environment variables
39688f1 
This diff continues refactoring how we set environment variables
to approximate what we were doing in bash scripts.

See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
refactor(buildtool): improve how we set environment variables (#1060)
38ebe0c 
This diff continues refactoring how we set environment variables to approximate what we were doing in bash in #1024.

See ooni/probe#2401
@bassosimone bassosimone mentioned this issue Jan 26, 2023
Merged
bassosimone added a commit to ooni/probe-cli that referenced this issue Jan 26, 2023
@bassosimone
feat: cross-compile tor for android using the Go builder (#1061)
2711d93 
This diff introduces the necessary changes for building tor and its dependencies using the Go builder. The reference issues are ooni/probe#2401 and ooni/probe#2365. A previous PR that used bash was #1024.

BTW, here's my sarcastic comment about doing this work in Go:

![cc07af6c-f63f-494c-a6df-02c1d12eab79_text](https://user-images.githubusercontent.com/337298/214907109-09a1264a-edc6-4b51-9114-8f405b9a3fb7.gif)

Ah, also: it was wrong of me to use `ANDROID_NDK_HOME` and I should have used `ANDROID_NDK_ROOT`. This pull request is a good moment in time to fix this blunder. While there, zap a README I should have zapped before. We keep them for top-level directories but not for nested directories, in general.
@bassosimone
Copy link
Contributor Author

All done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bassosimone bassosimone

Labels
enhancement improving existing code or new feature interrupt ooni/probe-cli ooni/probe-engine priority/high refactoring techdebt This issue describes technical debt
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bassosimone