Skip to content

Commit c527fad

Browse files
committed
Merge branch 'main' of github.com:linkall-labs/vanus into clientstream-v2
Signed-off-by: jyjiangkai <[email protected]>
2 parents 39a3140 + 7e5318d commit c527fad

File tree

15 files changed

+567
-381
lines changed

15 files changed

+567
-381
lines changed

README.md

Lines changed: 73 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,79 @@
1-
# Vanus
21

2+
<div align="center">
3+
<h1>Vanus</h1>
4+
<!--
5+
<img src="https://user-images.githubusercontent.capabilitiesom/68597908/206148625-43f14f58-f3c0-4042-82a0-9f9421c270fa.png" width="288" >
6+
-->
7+
<p>
8+
Vanus is an open-source message queue with built-in event processing capabilities.
9+
<!--
10+
<br>
11+
It helps users build event pipelines between SaaS, cloud services, and cloud functions in minutes.
12+
-->
13+
</p>
14+
15+
[![stars](https://img.shields.io/github/stars/linkall-labs/vanus.svg?style=flat&logo=github&colorB=blueviolet&label=stars)](https://github.com/linkall-labs/vanus)
316
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](https://github.com/linkall-labs/vanus/blob/main/LICENSE)
4-
[![Language](https://img.shields.io/badge/Language-Go-blue.svg)](https://golang.org/)
517
[![codecov](https://codecov.io/gh/linkall-labs/vanus/branch/main/graph/badge.svg?token=RSXSIMEY4V)](https://codecov.io/gh/linkall-labs/vanus)
6-
7-
## What is Vanus
8-
9-
Vanus is a Serverless event platform for easily building Event-Driven Architecture (EDA) applications.
10-
It provides enterprises and organizations an innovative platform for collecting, storing, distributing, and processing events at scale.
11-
12-
## Quick Start
13-
14-
See [quick-start](https://docs.linkall.com/getting-started/quick-start)
18+
[![Language](https://img.shields.io/github/go-mod/go-version/linkall-labs/vanus?logo=go)](https://golang.org/)
19+
[![playground](https://img.shields.io/badge/Playground-Try%20it%20%20free-red)](https://play.linkall.com)
20+
[![playground](https://img.shields.io/badge/Docs-online-green)](https://linkall.com/docs)
21+
22+
23+
<p>
24+
<a href="https://join.slack.com/t/vanusworkspace/shared_invite/zt-1jilbbfo2-NxiFG0VOo8ABGCCNaeNfcA"><img src="https://img.shields.io/badge/slack-join-3CC798?style=social&logo=slack" height=23></a>
25+
&nbsp;
26+
<a href="https://twitter.com/Vanus_dev"><img src="https://img.shields.io/badge/-Twitter-red?style=social&logo=twitter" height=23></a>
27+
<!-- <a href="https://twitter.com/Vanus_dev"><img src="https://img.shields.io/twitter/follow/vanus_dev?style=social" height=23></a> -->
28+
&nbsp;
29+
<a href="https://www.youtube.com/channel/UC7rd6IgjfNYTOXf2FerFNyA"><img src="https://img.shields.io/badge/-Youtube-red?style=social&logo=youtube" height=23></a>
30+
<!-- <a href="https://www.youtube.com/channel/UC7rd6IgjfNYTOXf2FerFNyA"><img src="https://img.shields.io/youtube/channel/views/UC7rd6IgjfNYTOXf2FerFNyA?style=social" height=23></a> -->
31+
&nbsp;
32+
</p>
33+
34+
<!--
35+
<h3 align="center">
36+
<a href="https://featbit.gitbook.io/docs/installation">Installation</a>
37+
<span> · </span>
38+
<a href="https://featbit.gitbook.io/">Getting Started</a>
39+
<span> · </span>
40+
<a href="https://join.slack.com/t/featbit/shared_invite/zt-1ew5e2vbb-x6Apan1xZOaYMnFzqZkGNQ">Online Support</a>
41+
<span> · </span>
42+
<a href="https://featbit.gitbook.io/">Documentation</a>
43+
<span> · </span>
44+
<a href="https://github.com/featbit/featbit/discussions/categories/announcements">Milestones</a>
45+
</h3>
46+
-->
47+
</div>
48+
49+
--------------------------------------------------
50+
51+
## Introduction
52+
53+
Vanus helps users build event pipelines between SaaS, cloud services, and cloud functions in minutes.
54+
55+
**1. Build the event-driven system**
56+
- Get events from cloud services and SaaS, and deliver them to cloud functions or microservices.
57+
- Synchronize changed data or transfer data to the data lake.
58+
- Obtain events generated by SaaS and send them to other SaaS.
59+
60+
**2. Out-of-the-box event computing capabilities**
61+
- Real-time processing during event transmissions, such as filtering and transformation.
62+
- Natively support the CloudEvents specification, and can directly send events to workloads that support CloudEvent.
63+
64+
**3. 100% open source, Super easy to use**
65+
- One-click deployment, the installation is completed within 1 minute, and developers without MQ experience can also use it.
66+
- Message queues and connectors are 100% open source, a one-stop open-source solution.
67+
68+
## Getting Started
69+
70+
You can install Vanus with a single command within 1 minute. Check out our [website](https://linkall.com) for detailed information.
71+
72+
```shell
73+
kubectl apply -f https://download.linkall.com/all-in-one/v0.5.4.yml
74+
```
75+
76+
We also provide an [interactive Kubernetes environment](https://play.linkall.com) to simply deploy and try Vanus in your browser.
1577

1678
## Community
1779

@@ -26,17 +88,6 @@ We have a few channels for contact:
2688

2789
See [here](CONTRIBUTING.md) for how to contribute to Vanus.
2890

29-
## Understanding Vanus
30-
31-
### Project Layout
32-
33-
This Project follows [golang-standards/project-layout](https://github.com/golang-standards/project-layout), see the
34-
project for understanding well vanus' codebase.
35-
36-
### Architecture
37-
38-
![architecture](docs/architecture.jpg)
39-
4091
## License
4192

4293
Vanus is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.

client/internal/vanus/store/block_store.go

Lines changed: 37 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ import (
1818
// standard libraries
1919
"context"
2020
"io"
21-
"math/rand"
2221
"sync"
22+
"sync/atomic"
2323
"time"
2424

2525
"github.com/linkall-labs/vanus/observability/tracing"
@@ -66,17 +66,18 @@ func newBlockStore(endpoint string) (*BlockStore, error) {
6666
return s, nil
6767
}
6868

69-
type streamstate string
69+
type streamState string
7070

7171
var (
72-
stateRunning streamstate = "running"
73-
stateCLosed streamstate = "closed"
72+
stateRunning streamState = "running"
73+
stateClosed streamState = "closed"
7474
)
7575

7676
type appendStreamCache struct {
77+
opaqueID uint64
7778
stream segpb.SegmentServer_AppendToBlockStreamClient
7879
callbacks sync.Map
79-
state streamstate
80+
state streamState
8081
once sync.Once
8182
}
8283

@@ -85,7 +86,7 @@ func (a *appendStreamCache) isRunning() bool {
8586
}
8687

8788
func (a *appendStreamCache) isClosed() bool {
88-
return a.state == stateCLosed
89+
return a.state == stateClosed
8990
}
9091

9192
func (a *appendStreamCache) release() {
@@ -96,27 +97,26 @@ func (a *appendStreamCache) release() {
9697
func (a *appendStreamCache) releaseStream() {
9798
a.once.Do(func() {
9899
a.stream.CloseSend()
99-
a.state = stateCLosed
100+
a.state = stateClosed
100101
})
101102
}
102103

103104
func (a *appendStreamCache) releaseCallbacks() {
104105
a.callbacks.Range(func(key, value interface{}) bool {
105-
if value != nil {
106-
value.(appendCallback)(&segpb.AppendToBlockStreamResponse{
107-
ResponseCode: errpb.ErrorCode_CLOSED,
108-
ResponseMsg: "append stream closed",
109-
Offsets: []int64{},
110-
})
111-
}
106+
value.(appendCallback)(&segpb.AppendToBlockStreamResponse{
107+
ResponseCode: errpb.ErrorCode_CLOSED,
108+
ResponseMsg: "append stream closed",
109+
Offsets: []int64{},
110+
})
112111
return true
113112
})
114113
}
115114

116115
type readStreamCache struct {
116+
opaqueID uint64
117117
stream segpb.SegmentServer_ReadFromBlockStreamClient
118118
callbacks sync.Map
119-
state streamstate
119+
state streamState
120120
once sync.Once
121121
}
122122

@@ -125,7 +125,7 @@ func (r *readStreamCache) isRunning() bool {
125125
}
126126

127127
func (r *readStreamCache) isClosed() bool {
128-
return r.state == stateCLosed
128+
return r.state == stateClosed
129129
}
130130

131131
func (r *readStreamCache) release() {
@@ -136,21 +136,19 @@ func (r *readStreamCache) release() {
136136
func (r *readStreamCache) releaseStream() {
137137
r.once.Do(func() {
138138
r.stream.CloseSend()
139-
r.state = stateCLosed
139+
r.state = stateClosed
140140
})
141141
}
142142

143143
func (r *readStreamCache) releaseCallbacks() {
144144
r.callbacks.Range(func(key, value interface{}) bool {
145-
if value != nil {
146-
value.(readCallback)(&segpb.ReadFromBlockStreamResponse{
147-
ResponseCode: errpb.ErrorCode_CLOSED,
148-
ResponseMsg: "read stream closed",
149-
Events: &cepb.CloudEventBatch{
150-
Events: []*cepb.CloudEvent{},
151-
},
152-
})
153-
}
145+
value.(readCallback)(&segpb.ReadFromBlockStreamResponse{
146+
ResponseCode: errpb.ErrorCode_CLOSED,
147+
ResponseMsg: "read stream closed",
148+
Events: &cepb.CloudEventBatch{
149+
Events: []*cepb.CloudEvent{},
150+
},
151+
})
154152
return true
155153
})
156154
}
@@ -335,7 +333,7 @@ func (s *BlockStore) AppendManyStream(ctx context.Context, block uint64, events
335333
}
336334

337335
// generate unique opaqueID
338-
opaqueID := rand.New(rand.NewSource(time.Now().UnixNano())).Uint64()
336+
atomic.AddUint64(&append.opaqueID, 1)
339337

340338
//TODO(jiangkai): delete the reference of CloudEvents/v2 in Vanus
341339
eventpbs := make([]*cepb.CloudEvent, len(events))
@@ -348,13 +346,13 @@ func (s *BlockStore) AppendManyStream(ctx context.Context, block uint64, events
348346
}
349347

350348
donec := make(chan struct{})
351-
append.callbacks.Store(opaqueID, appendCallback(func(res *segpb.AppendToBlockStreamResponse) {
349+
append.callbacks.Store(append.opaqueID, appendCallback(func(res *segpb.AppendToBlockStreamResponse) {
352350
resp = res
353351
close(donec)
354352
}))
355353

356354
req := &segpb.AppendToBlockStreamRequest{
357-
Id: opaqueID,
355+
Id: append.opaqueID,
358356
BlockId: block,
359357
Events: &cepb.CloudEventBatch{
360358
Events: eventpbs,
@@ -369,10 +367,10 @@ func (s *BlockStore) AppendManyStream(ctx context.Context, block uint64, events
369367
append.releaseStream()
370368
// reset new stream connections
371369
s.connectAppendStream(ctx)
372-
c, _ := append.callbacks.LoadAndDelete(opaqueID)
370+
c, _ := append.callbacks.LoadAndDelete(append.opaqueID)
373371
if c != nil {
374372
c.(appendCallback)(&segpb.AppendToBlockStreamResponse{
375-
Id: opaqueID,
373+
Id: append.opaqueID,
376374
ResponseCode: errpb.ErrorCode_CLOSED,
377375
ResponseMsg: "append stream closed",
378376
Offsets: []int64{},
@@ -384,10 +382,10 @@ func (s *BlockStore) AppendManyStream(ctx context.Context, block uint64, events
384382
select {
385383
case <-donec:
386384
case <-_ctx.Done():
387-
c, _ := append.callbacks.LoadAndDelete(opaqueID)
385+
c, _ := append.callbacks.LoadAndDelete(append.opaqueID)
388386
if c != nil {
389387
c.(appendCallback)(&segpb.AppendToBlockStreamResponse{
390-
Id: opaqueID,
388+
Id: append.opaqueID,
391389
ResponseCode: errpb.ErrorCode_CONTEXT_CANCELED,
392390
ResponseMsg: "append stream context canceled",
393391
Offsets: []int64{},
@@ -465,10 +463,10 @@ func (s *BlockStore) ReadStream(
465463
}
466464

467465
// generate unique RequestId
468-
opaqueID := rand.New(rand.NewSource(time.Now().UnixNano())).Uint64()
466+
atomic.AddUint64(&read.opaqueID, 1)
469467

470468
donec := make(chan struct{})
471-
read.callbacks.Store(opaqueID, readCallback(func(res *segpb.ReadFromBlockStreamResponse) {
469+
read.callbacks.Store(read.opaqueID, readCallback(func(res *segpb.ReadFromBlockStreamResponse) {
472470
resp = res
473471
close(donec)
474472
}))
@@ -486,10 +484,10 @@ func (s *BlockStore) ReadStream(
486484
})
487485
read.releaseStream()
488486
s.connectReadStream(ctx)
489-
c, _ := read.callbacks.LoadAndDelete(opaqueID)
487+
c, _ := read.callbacks.LoadAndDelete(read.opaqueID)
490488
if c != nil {
491489
c.(readCallback)(&segpb.ReadFromBlockStreamResponse{
492-
Id: opaqueID,
490+
Id: read.opaqueID,
493491
ResponseCode: errpb.ErrorCode_CLOSED,
494492
ResponseMsg: "read stream closed",
495493
Events: &cepb.CloudEventBatch{
@@ -503,10 +501,10 @@ func (s *BlockStore) ReadStream(
503501
select {
504502
case <-donec:
505503
case <-_ctx.Done():
506-
c, _ := read.callbacks.LoadAndDelete(opaqueID)
504+
c, _ := read.callbacks.LoadAndDelete(read.opaqueID)
507505
if c != nil {
508506
c.(readCallback)(&segpb.ReadFromBlockStreamResponse{
509-
Id: opaqueID,
507+
Id: read.opaqueID,
510508
ResponseCode: errpb.ErrorCode_CONTEXT_CANCELED,
511509
ResponseMsg: "read stream context canceled",
512510
Events: &cepb.CloudEventBatch{

internal/store/block/block.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,10 @@ type Reader interface {
4444
Read(ctx context.Context, seq int64, num int) ([]Entry, error)
4545
}
4646

47+
type AppendCallback = func(seqs []int64, err error)
48+
4749
type Appender interface {
48-
Append(ctx context.Context, entries []Entry, cb func([]int64, error))
50+
Append(ctx context.Context, entries []Entry, cb AppendCallback)
4951
}
5052

5153
type Block interface {

0 commit comments

Comments
 (0)