Skip to content

Commit 61af770

Browse files
vweeversvweevers
committed
Rename type property to name
Because `type` is too close to `format` and `name` gives us compatibility with various other modules.
1 parent e3087a5 commit 61af770

13 files changed

+332
-194
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
node_modules
22
.nyc_output/
3+
coverage/

‎README.md

Lines changed: 51 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# level-transcoder
22

3-
**Encode data with built-in or custom encodings.** The (not yet official) successor to `level-codec`, that introduces "transcoders" to translate between encodings and internal data formats supported by a db. This allows a db to store keys and values in a format of its choice (Buffer, Uint8Array or String) with zero-effort support of all known encodings.
3+
**Encode data with built-in or custom encodings.** The (not yet official) successor to [`level-codec`][level-codec] that introduces "transcoders" to translate between encodings and internal data formats supported by a db. This allows a db to store keys and values in a format of its choice (Buffer, Uint8Array or String) with zero-effort support of all known encodings.
44

55
[![level badge][level-badge]](https://github.com/Level/awesome)
66
[![Test](https://img.shields.io/github/workflow/status/Level/transcoder/Test?label=test)](https://github.com/Level/transcoder/actions/workflows/test.yml)
@@ -11,8 +11,6 @@
1111

1212
## Usage
1313

14-
**This is in POC stage.**
15-
1614
```js
1715
const Transcoder = require('level-transcoder')
1816

@@ -31,7 +29,7 @@ console.log(transcoder2.encoding('json').encode(123))
3129
console.log(transcoder3.encoding('json').encode(123))
3230
```
3331

34-
If given multiple formats (like how `leveldown` can work with both Buffer and strings), the best fitting format is chosen. Not by magic, just hardcoded logic because we don't have that many formats to deal with.
32+
If given multiple formats (like how [`leveldown`][leveldown] can work with both Buffer and strings), the best fitting format is chosen. Not by magic, just hardcoded logic because we don't have that many formats to deal with.
3533

3634
For example, knowing that JSON is a UTF-8 string which matches the desired `utf8` format, the `json` encoding will return a string here:
3735

@@ -53,7 +51,29 @@ Copying of data is avoided where possible. That's true in the last example, beca
5351

5452
Lastly, the encoding returned by `Transcoder#encoding()` has a `format` property to be used to forward key- and valueEncoding options to an underlying store. This way, both the public and private API's of a db will be encoding-aware (somewhere in the future).
5553

56-
For example, on `leveldown` a call like `db.put(key, 123, { valueEncoding: 'json' })` will pass that value `123` through a `json` encoding that has a `format` of `utf8`, which is then forwarded as `db._put(key, '123', { valueEncoding: 'utf8' })`.
54+
For example, on `leveldown` a call like `db.put(key, { x: 3 }, { valueEncoding: 'json' })` will pass that value `{ x: 3 }` through a `json` encoding that has a `format` of `utf8`, which is then forwarded as `db._put(key, '{"x":3}', { valueEncoding: 'utf8' })`.
55+
56+
## Compatible with
57+
58+
Various modules in the ecosystem, in and outside of level, can be used with `level-transcoder`.
59+
60+
| Module | Format | Interface | Named |
61+
|:-------------------------------------------|:-------------|:-----------------------------|:------|
62+
| [`protocol-buffers`][protocol-buffers] | buffer | `level-codec` | ❌ |
63+
| [`charwise`][charwise] | utf8 | `level-codec` | ✅ |
64+
| [`bytewise`][bytewise] | buffer | `level-codec` | ✅ |
65+
| [`lexicographic-integer-encoding`][lexint] | buffer, utf8 | `level-codec` | ✅ |
66+
| [`codecs`][mafintosh-codecs] | buffer | `codecs` | ✅ |
67+
| [`abstract-encoding`][abstract-encoding] | buffer | `abstract-encoding` | ❌ |
68+
| [`multiformats`][js-multiformats] | view | [`multiformats`][blockcodec] | ✅ |
69+
| [`base32-codecs`][base32-codecs] | buffer | `codecs` | ✅ |
70+
| [`level-codec`][level-codec] | buffer, utf8 | `level-codec` | ✅ |
71+
72+
Common between the interfaces is that they have `encode()` and `decode()` methods. The terms "codec" and "encoding" are used interchangeably. Passing these encodings through `Transcoder#encoding()` (which is done implicitly when used in an `abstract-level` database) results in normalized encoding objects that follow [the interface](./lib/encoding.d.ts) of `level-transcoder`.
73+
74+
If the format in the table above is buffer, then `encode()` is expected to return a Buffer. If utf8, then a string. If view, then a Uint8Array.
75+
76+
Those marked as not named are modules that export or generate anonymous encodings that don't have a `name` property (or `type` as an alias) which means they can only be used as objects and not by name. Passing an anonymous encoding through `Transcoder#encoding()` does give it a `name` property for compatibility, but the value of `name` is not deterministic.
5777

5878
---
5979

@@ -123,7 +143,7 @@ See below for a list and the format of `encoding`.
123143
| `hex`<br>`ascii`<br>`base64`<br>`ucs2`<br>`utf16le`<br>`utf-16le` | String or Buffer | Buffer | String |
124144
| `none` a.k.a. `id` | Any type (bypass encoding) | Input\* | As stored |
125145

126-
<sup>\*</sup> Stores may have their own type coercion. Whether type information is preserved depends on the [`abstract-leveldown`] implementation as well as the underlying storage (`LevelDB`, `IndexedDB`, etc).
146+
<sup>\*</sup> Stores may have their own type coercion. Whether type information is preserved depends on the [`abstract-leveldown`][abstract-leveldown] implementation as well as the underlying storage (`LevelDB`, `IndexedDB`, etc).
127147

128148
## Encoding Format
129149

@@ -146,7 +166,7 @@ All of these properties are required.
146166

147167
The `buffer` boolean tells consumers whether to fetch data as a Buffer, before calling your `decode()` function on that data. If `buffer` is true, it is assumed that `decode()` takes a Buffer. If false, it is assumed that `decode` takes any other type (usually a string).
148168

149-
To explain this in the grand scheme of things, consider a store like [`leveldown`] which has the ability to return either a Buffer or string, both sourced from the same byte array. Wrap this store with [`encoding-down`] and it'll select the most optimal data type based on the `buffer` property of the active encoding. If your `decode()` function needs a string (and the data can legitimately become a UTF8 string), you should set `buffer` to `false`. This avoids the cost of having to convert a Buffer to a string.
169+
To explain this in the grand scheme of things, consider a store like [`leveldown`][leveldown] which has the ability to return either a Buffer or string, both sourced from the same byte array. Wrap this store with [`encoding-down`][encoding-down] and it'll select the most optimal data type based on the `buffer` property of the active encoding. If your `decode()` function needs a string (and the data can legitimately become a UTF8 string), you should set `buffer` to `false`. This avoids the cost of having to convert a Buffer to a string.
150170

151171
The `type` string should be a unique name.
152172

@@ -168,8 +188,29 @@ Support us with a monthly donation on [Open Collective](https://opencollective.c
168188

169189
[level-badge]: https://leveljs.org/img/badge.svg
170190

171-
[`encoding-down`]: https://github.com/Level/encoding-down
191+
[level-codec]: https://github.com/Level/codec
192+
193+
[encoding-down]: https://github.com/Level/encoding-down
194+
195+
[abstract-leveldown]: https://github.com/Level/abstract-leveldown
196+
197+
[leveldown]: https://github.com/Level/leveldown
198+
199+
[protocol-buffers]: https://github.com/mafintosh/protocol-buffers
200+
201+
[charwise]: https://github.com/dominictarr/charwise
202+
203+
[bytewise]: https://github.com/deanlandolt/bytewise
204+
205+
[lexint]: https://github.com/vweevers/lexicographic-integer-encoding
206+
207+
[mafintosh-codecs]: https://github.com/mafintosh/codecs
208+
209+
[abstract-encoding]: https://github.com/mafintosh/abstract-encoding
210+
211+
[js-multiformats]: https://github.com/multiformats/js-multiformats
212+
213+
[blockcodec]: https://github.com/multiformats/js-multiformats/blob/master/src/codecs/interface.ts
172214

173-
[`abstract-leveldown`]: https://github.com/Level/abstract-leveldown
215+
[base32-codecs]: https://github.com/consento-org/base32-codecs
174216

175-
[`leveldown`]: https://github.com/Level/leveldown

‎index.d.ts

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,9 @@ declare class Transcoder<T = any> {
88
constructor (formats: Iterable<string>)
99

1010
/**
11-
* Get the types of supported encodings.
12-
* @param full Return fully qualified types with the formats of trancoded encodings (if any).
11+
* Get supported encoding objects.
1312
*/
14-
types (full?: boolean): string[]
13+
encodings (): Array<Encoding<any, T, any>>
1514

1615
/**
1716
* Get the given encoding, creating a transcoder if necessary.

‎index.js

Lines changed: 52 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -25,15 +25,15 @@ class Transcoder {
2525
this[kFormats] = new Set(formats)
2626

2727
// Only support aliases in key- and valueEncoding options (where we already did)
28-
for (const [alias, { type }] of Object.entries(aliases)) {
28+
for (const [alias, { name }] of Object.entries(aliases)) {
2929
if (this[kFormats].has(alias)) {
30-
throw new ModuleError(`The '${alias}' alias is not supported here; use '${type}' instead`, {
30+
throw new ModuleError(`The '${alias}' alias is not supported here; use '${name}' instead`, {
3131
code: 'LEVEL_ENCODING_NOT_SUPPORTED'
3232
})
3333
}
3434
}
3535

36-
// Register encodings (done early in order to populate types())
36+
// Register encodings (done early in order to populate encodings())
3737
for (const k in encodings) {
3838
try {
3939
this.encoding(k)
@@ -45,16 +45,10 @@ class Transcoder {
4545
}
4646

4747
/**
48-
* @param {boolean} [full]
48+
* @returns {Array<Encoding<any,T,any>>}
4949
*/
50-
types (full) {
51-
const types = new Set()
52-
53-
for (const encoding of this[kEncodings].values()) {
54-
types.add(full ? encoding.type : encoding.type.split('+')[0])
55-
}
56-
57-
return Array.from(types)
50+
encodings () {
51+
return Array.from(new Set(this[kEncodings].values()))
5852
}
5953

6054
/**
@@ -69,39 +63,33 @@ class Transcoder {
6963
resolved = lookup[encoding]
7064

7165
if (!resolved) {
72-
throw new ModuleError(
73-
`Encoding '${encoding}' is not found`,
74-
{ code: 'LEVEL_ENCODING_NOT_FOUND' }
75-
)
66+
throw new ModuleError(`Encoding '${encoding}' is not found`, {
67+
code: 'LEVEL_ENCODING_NOT_FOUND'
68+
})
7669
}
7770
} else if (typeof encoding !== 'object' || encoding === null) {
7871
throw new TypeError("First argument 'encoding' must be a string or object")
7972
} else if (encoding instanceof Encoding) {
8073
resolved = encoding
81-
} else if (encoding.format === 'view') {
82-
resolved = new ViewFormat(encoding)
83-
} else if (encoding.format === 'utf8' || encoding.buffer === false) {
84-
resolved = new UTF8Format(encoding)
8574
} else {
86-
resolved = new BufferFormat(encoding)
75+
resolved = from(encoding)
8776
}
8877

89-
const { type, format } = resolved
78+
const { name, format } = resolved
9079

9180
if (!this[kFormats].has(format)) {
9281
if (this[kFormats].has('view')) {
93-
resolved = resolved.transcode('view')
82+
resolved = resolved.createViewTranscoder()
9483
} else if (this[kFormats].has('buffer')) {
95-
resolved = resolved.transcode('buffer')
84+
resolved = resolved.createBufferTranscoder()
9685
} else {
97-
// TODO: improve error message (see tests, it's inconsistent)
98-
throw new ModuleError(`Encoding '${type}' is not supported`, {
86+
throw new ModuleError(`Encoding '${name}' is not supported`, {
9987
code: 'LEVEL_ENCODING_NOT_SUPPORTED'
10088
})
10189
}
10290
}
10391

104-
for (const k of [encoding, type, resolved.type]) {
92+
for (const k of [encoding, name, resolved.name, resolved.commonName]) {
10593
this[kEncodings].set(k, resolved)
10694
}
10795
}
@@ -112,6 +100,43 @@ class Transcoder {
112100

113101
module.exports = Transcoder
114102

103+
/**
104+
* @param {EncodingOptions<any, any, any>} options
105+
* @returns {Encoding<any, any, any>}
106+
*/
107+
function from (options) {
108+
const format = detectFormat(options)
109+
110+
switch (format) {
111+
case 'view': return new ViewFormat(options)
112+
case 'utf8': return new UTF8Format(options)
113+
case 'buffer': return new BufferFormat(options)
114+
default: {
115+
throw new ModuleError(`Encoding '${format}' is not supported`, {
116+
code: 'LEVEL_ENCODING_NOT_SUPPORTED'
117+
})
118+
}
119+
}
120+
}
121+
122+
/**
123+
* If format is not provided, fallback to detecting `level-codec`
124+
* or `multiformats` encodings, else assume a format of buffer.
125+
* @param {EncodingOptions<any, any, any>} options
126+
* @returns {string}
127+
*/
128+
function detectFormat ({ format, buffer, code }) {
129+
if (format !== undefined) {
130+
return format
131+
} else if (typeof buffer === 'boolean') {
132+
return buffer ? 'buffer' : 'utf8' // level-codec
133+
} else if (Number.isInteger(code)) {
134+
return 'view' // multiformats
135+
} else {
136+
return 'buffer'
137+
}
138+
}
139+
115140
/**
116141
* @typedef {import('./lib/encoding').EncodingOptions<TIn,TFormat,TOut>} EncodingOptions
117142
* @template TIn, TFormat, TOut

‎lib/encoding.d.ts

Lines changed: 37 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -14,23 +14,30 @@ export abstract class Encoding<TIn, TFormat, TOut> {
1414
decode: (data: TFormat) => TOut
1515

1616
/** Unique name for this encoding. */
17-
type: string
17+
name: string
1818

1919
/**
20-
* Indicates the (lower-level) encoding used by the return value
21-
* of {@link encode}. Typically one of 'buffer', 'view', 'utf8'.
20+
* Common name. If this encoding is a transcoder, {@link name} will be for
21+
* example 'json+view' and {@link commonName} will be just 'json'. Else
22+
* {@link name} will equal {@link commonName}.
23+
*/
24+
get commonName (): string
25+
26+
/**
27+
* The name of the (lower-level) encoding used by the return value of
28+
* {@link encode}. Typically one of 'buffer', 'view', 'utf8'.
2229
*/
2330
format: string
2431

25-
createViewTranscoder (): ViewFormat<TIn, TOut> | undefined
26-
createBufferTranscoder (): BufferFormat<TIn, TOut> | undefined
32+
/**
33+
* Create a new encoding that transcodes {@link TFormat} to a view.
34+
*/
35+
createViewTranscoder (): ViewFormat<TIn, TOut>
2736

2837
/**
29-
* Create a new encoding that transcodes from this to {@link format}.
30-
* @param format What to encode to.
38+
* Create a new encoding that transcodes {@link TFormat} to a buffer.
3139
*/
32-
transcode (format: 'view'): ViewFormat<TIn, TOut>
33-
transcode (format: 'buffer'): BufferFormat<TIn, TOut>
40+
createBufferTranscoder (): BufferFormat<TIn, TOut>
3441
}
3542

3643
export interface EncodingOptions<TIn, TFormat, TOut> {
@@ -47,20 +54,36 @@ export interface EncodingOptions<TIn, TFormat, TOut> {
4754
/**
4855
* Unique name for this encoding.
4956
*/
50-
type?: string | undefined
57+
name?: string | undefined
5158

5259
/**
53-
* Indicates the (lower-level) encoding used by the return value
54-
* of {@link encode}. Typically one of 'buffer', 'view', 'utf8'.
60+
* The name of the (lower-level) encoding used by the return value of
61+
* {@link encode}. Typically one of 'buffer', 'view', 'utf8'. Defaults to
62+
* 'buffer' if the {@link buffer} and {@link code} options are also undefined.
5563
*/
5664
format?: string | undefined
5765

5866
/**
59-
* Legacy property that means the same as `format: 'buffer'`.
60-
* Used only when `format` is not provided.
67+
* Legacy `level-codec` option that means the same as `format: 'buffer'`
68+
* if true or `format: 'utf8'` if false. Used only when the {@link format} option
69+
* is undefined.
6170
*/
6271
buffer?: boolean | undefined
6372

73+
/**
74+
* Legacy `level-codec` alias for {@link name}. Used only when the
75+
* {@link name} option is undefined.
76+
*/
77+
type?: any
78+
79+
/**
80+
* For compatibility with `multiformats`. If a number, then the encoding is
81+
* assumed to have a {@link format} of 'view'. Used only when the {@link format}
82+
* and {@link buffer} options are undefined.
83+
* @see https://github.com/multiformats/js-multiformats/blob/master/src/codecs/interface.ts
84+
*/
85+
code?: any
86+
6487
createViewTranscoder?: (() => ViewFormat<TIn, TOut>) | undefined
6588
createBufferTranscoder?: (() => BufferFormat<TIn, TOut>) | undefined
6689
}

0 commit comments

Comments
 (0)