Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProtoBuf 1.0.0 #202

Merged
merged 66 commits into from
Aug 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
9bc5c07
Initial commit
Drvi Jun 7, 2022
3e65f18
Module generation progress
Drvi Jun 9, 2022
e69c6e1
Towards handling imported type references
Drvi Jun 9, 2022
ef52295
Handle self-referential structs
Drvi Jun 10, 2022
db2010e
Rudimentary decoding codegen
Drvi Jun 15, 2022
cf13b60
Simplify decoding api, better prealocation
Drvi Jun 16, 2022
69ae1c2
Encoding progress
Drvi Jun 17, 2022
c29b245
Improve buffering
Drvi Jun 18, 2022
80613e4
Buffered decoding, more robust handling of repeated fields
Drvi Jun 19, 2022
4f17f12
JET and Aqua pass
Drvi Jun 20, 2022
3c51a3f
Merge pull request #1 from Drvi/td-init-decoding-codegen
Drvi Jun 21, 2022
400ffe5
Reuse topological sort code
Drvi Jun 21, 2022
cb7a8a5
Fix string lexing edge cases
Drvi Jun 21, 2022
6df5bd2
Codegen tweaks
Drvi Jun 21, 2022
8709d33
Remove startbyte and endbyte from lexer
Drvi Jun 22, 2022
f32c61c
Split codegen to multiple files
Drvi Jun 22, 2022
01ddac2
Towards more robust handling of referenced types
Drvi Jun 22, 2022
8846b33
Vbyte tests
Drvi Jun 23, 2022
953a953
Fix topological sort
Drvi Jun 23, 2022
032d66f
More "linker" improvements
Drvi Jun 23, 2022
61df428
More tests for codecs
Drvi Jun 23, 2022
db50814
Fix map codecs dispatch on Val{...}
Drvi Jun 23, 2022
55e7147
Dict encoding adds metadata per element
Drvi Jun 24, 2022
9ef1cce
Dict decoding + repeated map values support
Drvi Jun 24, 2022
b2268fc
Add TranscodingStreams
Drvi Jun 25, 2022
2d61a16
Fix oneof encode method generation
Drvi Jun 26, 2022
69a383a
Fix enum encoding/decoding
Drvi Jun 26, 2022
45cd7f4
Simple message test with oneof
Drvi Jun 27, 2022
ff0bd8e
Tweak decode method generation
Drvi Jun 27, 2022
bab1818
Test for basic message fields encoding/decoding
Drvi Jun 27, 2022
cb73f01
Introduce LengthDelimitedProtoDecoder to handle nested messages
Drvi Jun 28, 2022
c899108
Fix typo
Drvi Jun 28, 2022
a8f7e12
Test repeated length delimited fields
Drvi Jun 28, 2022
9b27d6d
Fix typo in decoder skip method
Drvi Jun 28, 2022
0f621f4
Use a faster string deserialization by @quinnj
Drvi Jun 28, 2022
1dfe2d3
Clean up OneOf, add TODOs
Drvi Jun 28, 2022
9bdcaec
Fix repeated length delimited fields tests
Drvi Jun 28, 2022
61d4493
More TODOs
Drvi Jun 28, 2022
5846ac9
Codecs tweaks for repeated messages
Drvi Jun 29, 2022
781f191
Add compat bounds for TranscodingStreams
Drvi Jun 29, 2022
d02c8b8
Tweak merging of structs
Drvi Jun 30, 2022
002d4f2
Enums in proto3 must start with a zero
Drvi Jun 30, 2022
5b7f4c0
No syntax specifier means proto2
Drvi Jun 30, 2022
21a1993
Stricted Enum parsing
Drvi Jun 30, 2022
3f21ec3
Add basic metadata providing methods in codegen
Drvi Jun 30, 2022
e544995
Enums don't carry reserved and extensions
Drvi Jul 1, 2022
6eacf6f
First draft of generate_oneof_fields_metadata_method
Drvi Jul 1, 2022
f8d2f7c
Revamped metadata methods
Drvi Jul 1, 2022
b7b2e73
Setup protojl tests
Drvi Jul 1, 2022
c504c5d
Setup CI (#3)
Drvi Jul 4, 2022
a7ac0e5
Add a `always_use_modules` option to `protojl` (#2)
Drvi Jul 11, 2022
bf0b1c4
Make struct fields nullable by default (with an override) (#4)
Drvi Jul 12, 2022
caa3cf5
Add generic fallback to metadata methods (#5)
Drvi Jul 13, 2022
bbb094a
Add `add_kwarg_constructors` option & default value tests (#6)
Drvi Jul 14, 2022
15f5cf1
Optimization pass (#7)
Drvi Jul 21, 2022
d464ab7
Add `parametrize_oneofs` option to `protojl` (#8)
Drvi Jul 22, 2022
370d32c
Various refactors that reduce compilation latency (#9)
Drvi Jul 22, 2022
dc599cf
Misc improvements (#10)
Drvi Jul 23, 2022
ae2deb2
Init docs (#11)
Drvi Jul 26, 2022
4d28123
Fix example
Drvi Jul 26, 2022
95c669d
More robust module generation & dependency handling (#12)
Drvi Aug 13, 2022
924b661
Allow leaf module to have the same name as top module (#17)
Drvi Aug 15, 2022
034270a
Fix for containers of parametric types (#15)
Drvi Aug 15, 2022
16ac873
Merge remote-tracking branch 'ProtocolBuffers.jl/master'
Drvi Aug 16, 2022
28ac2bc
Merge remote-tracking branch 'ProtocolBuffers.jl/master'
Drvi Aug 16, 2022
79f2eaf
Post-merge cleanup
Drvi Aug 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Run benchmarks

on:
pull_request:
types: [labeled, opened, synchronize, reopened]

# Only trigger the benchmark job when you add `run benchmark` label to the PR
jobs:
Benchmark:
runs-on: ubuntu-latest
if: contains(github.event.pull_request.labels.*.name, 'run benchmark')
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: 1
- uses: julia-actions/julia-buildpkg@latest
- name: Install dependencies
run: julia --color=yes -e 'using Pkg; pkg"add PkgBenchmark [email protected]"'
- name: Run benchmarks
run: julia --color=yes -e 'using BenchmarkCI; BenchmarkCI.judge()'
- name: Print judgement
run: julia --color=yes -e 'using BenchmarkCI; BenchmarkCI.displayjudgement()'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
46 changes: 28 additions & 18 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
name: CI
on:
push:
branches: [master]
tags: ["*"]
pull_request:
branches:
- master
push:
branches:
- master
tags: '*'
jobs:
test:
name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
Expand All @@ -12,25 +15,14 @@ jobs:
fail-fast: false
matrix:
version:
- '1.3'
- '1' # automatically expands to the latest stable 1.x release of Julia
- nightly
- '1'
- 'nightly'
os:
- ubuntu-latest
- macOS-latest
- windows-latest
arch:
- x64
- x86
include:
# test macOS and Windows with latest Julia only
- os: macOS-latest
arch: x64
version: 1
- os: windows-latest
arch: x64
version: 1
- os: windows-latest
arch: x86
version: 1
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@v1
Expand All @@ -53,3 +45,21 @@ jobs:
- uses: codecov/codecov-action@v1
with:
file: lcov.info

docs:
name: Documentation
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@v1
with:
version: '1'
- run: |
julia --project=docs -e '
using Pkg
Pkg.develop(PackageSpec(path=pwd()))
Pkg.instantiate()'
- run: julia --project=docs docs/make.jl
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
9 changes: 8 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
Manifest.toml
.vscode
dev/
test/test_protos/
docs/build/
docs/site/
*.jl.cov
*.jl.*.cov
*.jl.mem
out
.DS_Store
2 changes: 1 addition & 1 deletion LICENSE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
The ProtoBuf.jl package is licensed under the MIT "Expat" License:

> Copyright (c) 2014-22: Julia Computing, Tanmay Mohapatra, and contributors
> Copyright (c) 2022: RelationalAI, Tomáš Drvoštěp, and contributors
>
> Permission is hereby granted, free of charge, to any person obtaining
> a copy of this software and associated documentation files (the
Expand Down
82 changes: 0 additions & 82 deletions PROTOC.md

This file was deleted.

23 changes: 14 additions & 9 deletions Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,26 @@ uuid = "3349acd9-ac6a-5e09-bcdb-63829b23a429"
keywords = ["protobuf", "protoc"]
license = "MIT"
desc = "Julia protobuf implementation"
version = "0.11.5"
authors = ["Tomáš Drvoštěp <[email protected]>"]
version = "1.0.0"

[deps]
Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
protoc_jll = "c7845625-083e-5bbe-8504-b32d602b7110"
BufferedStreams = "e1450e63-4bb3-523b-b2a4-4ffa8c0fd77d"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
EnumX = "4e289a0a-7415-4d19-859d-a7e5c4648b56"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
TranscodingStreams = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"

[compat]
julia = "1.3"
protoc_jll = "3"
BufferedStreams = "1.1"
EnumX = "1"
TranscodingStreams = "0.9"
julia = "1.7"

[extras]
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
Sockets = "6462fe0b-24de-5631-8697-dd941f90decc"
Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
JET = "c3a54625-cd67-489e-a8e7-0a5a0ff4e31b"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Serialization", "Test", "Random", "Sockets"]
test = ["Aqua", "JET", "Test"]
54 changes: 47 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,54 @@
# ProtoBuf.jl

[![Build Status](https://github.com/JuliaIO/ProtoBuf.jl/workflows/CI/badge.svg)](https://github.com/JuliaIO/ProtoBuf.jl/actions?query=workflow%3ACI+branch%3Amaster)
[![][docs-dev-img]][docs-dev-url]

[![codecov.io](http://codecov.io/github/JuliaIO/ProtoBuf.jl/coverage.svg?branch=master)](http://codecov.io/github/JuliaIO/ProtoBuf.jl?branch=master)
This is a Julia package that provides a compiler and a codec for Protocol Buffers.

[**Protocol buffers**](https://developers.google.com/protocol-buffers/docs/overview) are a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more.
[Protocol Buffers](https://developers.google.com/protocol-buffers) are a language-neutral, platform-neutral extensible mechanism for serializing structured data.

**ProtoBuf.jl** is a Julia implementation for protocol buffers.
## Example

- [Generating Julia Code from .proto Specifications](PROTOC.md)
- [Using ProtoBuf to Read and Write Messages](USAGE.md)
Given a `test.proto` file in your current working directory:
```protobuf
syntax = "proto3";

Both version 2 and 3 of the protobuf specification language are supported.
message MyMessage {
sint32 a = 1;
repeated string b = 2;
}
```
You can generate Julia bindings with the `protojl` function:
```julia
using ProtoBuf
protojl("test.proto", ".", "output_dir")
```

This will create a Julia file at `output_dir/test_pb.jl` which you can simply `include` and start using it to encode and decode messages:

```julia
include("output_dir/test_pb.jl")
# Main.test_pb

io = IOBuffer();

e = ProtoEncoder(io);

encode(e, test_pb.MyMessage(-1, ["a", "b"]))
# 8

seekstart(io);

d = ProtoDecoder(io);

decode(d, test_pb.MyMessage)
# Main.test_pb.MyMessage(-1, ["a", "b"])
```
## Acknowledgement

We'd like to thank the authors of the following packages, as we took inspiration from their projects:

* We used [Tokenize.jl](https://github.com/JuliaLang/Tokenize.jl) as a reference when implementing the Lexer/Parser.
* We used the pre-1.0 version of `ProtoBuf.jl` as a giant shoulder to stand on:).

[docs-dev-img]: https://img.shields.io/badge/docs-dev-blue.svg
[docs-dev-url]: https://drvi.github.io/ProtoBuf.jl/dev/
Loading