BigQueue-go

BigQueue-go is pure Golang implementation for big, fast and persistent queue based on memory mapped file. Its file storage structures is totally compatible with [BigQueue](https://github.com/bulldog2011/bigqueue)

Feature Highlight:

1. Fast: close to the speed of direct memory access, both enqueue and dequeue are close to O(1) memory access.
2. Big: the total size of the queue is only limited by the available disk space.
3. Persistent: all data in the queue is persisted on disk, and is crash resistant.
4. Reliable: OS will be responsible to presist the produced messages even your process crashes.
5. Realtime: messages produced by producer threads will be immediately visible to consumer threads.
6. Memory-efficient: automatic paging & swapping algorithm, only most-recently accessed data is kept in memory.
7. Thread-safe: multiple threads can concurrently enqueue and dequeue without data corruption.
8. Simple&Light : pure Golang implements without any 3rd-party library

Quick Start

Installing

To start using BigQueue-Go, install Go and run go get:

$ go get github.com/jhunters/bigqueue

To run testcases:

$ go test -v .

Importing bigqueue

To use bigqueue as an file implements queue, import as:

import	"github.com/jhunters/bigqueue"

func main() {
	var queue = new(bigqueue.FileQueue)

	err := queue.Open(".", "testqueue", nil)

	if err != nil {
		fmt.Println(err)
		return
	}
	defer queue.Close()
	
	data := []byte("hello jhunters")
	
	i, err := queue.Enqueue(data)
	if err != nil {
		fmt.Println(err)
		return
	} else {
		fmt.Println("Enqueued index=", i, string(data))
	}
	
	index, bb, err := queue.Dequeue()
	if err != nil {
		fmt.Println(err)
		return
	}
	
	fmt.Println("Dequeue data:", index, string(bb))
}

Docs

1. big queue tutorial
2. fanout queue tutorial

The Big Picture

Benmark test

$ go test -bench . -benchtime=3s -run=^$

goos: linux
goarch: amd64
pkg: github.com/bigqueue
Benchmark_EnqueueOnly-8                  2319403              1479 ns/op
Benchmark_DequeueOnly-8                  4704715               743 ns/op
Benchmark_EnqueueDequeue-8               1536244              2303 ns/op
Benchmark_ParallelEnqueueDequeue-8       1254315              2760 ns/op
PASS
ok      github.com/bigqueue     40.028s

License

BigQueue-Go is Apache 2.0 licensed.