Initial import, still WIP #4

juga0 · 2018-04-19T21:11:18Z

With changes after @teor2345 review on tor-dev ml.
Unfortunately lost all the granular commits changing line by line :/
I can rebase and make granular commits if need.
This PR is mostly to be reviewed by @teor2345 as i itwould be more difficult in ml, at least for all the format related changes.

With changes after T. review on tor-dev ml. Unfortunately lost all the granular ci changing line by line :/

teor2345

Looks great!
Let's make these minor tweaks, then send it to tor-dev for review again.

Please open a ticket on trac for merging this spec into torspec.

teor2345 · 2018-04-20T01:13:52Z

proposals/XX-bw-xxx.txt

@@ -0,0 +1,169 @@
+          Tor Bandwidth Measurements Document Format
+              juga


Nitpick: center name using spaces?

teor2345 · 2018-04-20T01:15:59Z

proposals/XX-bw-xxx.txt

+  - Header (exactly once)
+  - Relays measurements (zero or more times)
+
+2.1. Nonterminals


You could call this "Definitions"

teor2345 · 2018-04-20T01:16:14Z

proposals/XX-bw-xxx.txt

+
+    [At start, exactly once.]
+
+    timestamp = Int


You could define timestamp in the definitions section

But say what time it represents here.

should i use "=" or "::=" for the definitions?

teor2345 · 2018-04-20T01:19:14Z

proposals/XX-bw-xxx.txt

+  If a header line does conform to this format, the line SHOULD be ignored by
+  parsers.
+
+  [juga: how the header ends?, just when a relay measurement is found?]


Yes, sounds good to me.

I assume that's how Tor does it.

teor2345 · 2018-04-20T01:19:58Z

proposals/XX-bw-xxx.txt

+    Documents generated before this specification do not contain this line,
+    and the version_number is considered to be "1.0.0".
+    This line is ignored by any Tor version released before this specification,
+    April 2018.


We need to implement this feature in Tor, and then put the Tor version that implements the feature in this spec.
We can just put "TODO" for now.

See https://trac.torproject.org/projects/tor/ticket/3723

teor2345 · 2018-04-20T01:23:40Z

proposals/XX-bw-xxx.txt

+
+  Where:
+
+    timestamp = Int


You can define timestamp in the definitions section, and say what time it represents here.

teor2345 · 2018-04-20T01:24:45Z

proposals/XX-bw-xxx.txt

+
+    rtt = Int
+
+    The Round Trip Time to obtain 1 Byte of data.


In milliseconds?

teor2345 · 2018-04-20T01:25:12Z

proposals/XX-bw-xxx.txt

+
+    timestamp = Int
+
+    The Unix Epoch time when the measurement was performed.


teor2345 · 2018-04-20T01:25:24Z

proposals/XX-bw-xxx.txt

+
+    timestamp = Int
+
+    The Unix Epoch time when the file was created.


In seconds.

teor2345 · 2018-04-20T01:27:09Z

proposals/XX-bw-xxx.txt

+
+    The Round Trip Time to obtain 1 Byte of data.
+
+  [juga: which arguments should be include from torflow?]


Please include node_id, bw, nick.
I don't think there is a lot of value in documenting legacy systems.

Also, there is an example in the appendix, and you have referenced the documentation.

teor2345 · 2018-04-20T08:05:07Z

proposals/XX-bw-xxx.txt

+    and the version_number is considered to be "1.0.0".
+    This line is ignored by any Tor version released before this specification,
+    April 2018.
+


Can you add software and software_version lines as well?

For example:

software=sbws software_version=0.1.0

The default software should be torflow and there should be no default version.

Only in the sample data or in the definition/format sections too?

k, i answer myself, should be in definition/format sections too

teor2345 · 2018-04-20T22:38:41Z

On 21 Apr 2018, at 02:42, juga0 ***@***.***> wrote: should i use "=" or "::=" for the definitions?

dir-spec uses both, but mostly "::=". https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n212 So let's pick "::=" and be consistent. Using "::=" also makes it easier to tell the difference between definitions: version ::= Int "." Int "." Int ArgumentChar* and declarations: "software_version=" version

teor2345 · 2018-04-21T00:31:11Z

proposals/XX-bw-xxx.txt

+
+    New implementations SHOULD include this line.
+    For documents generated following this specification,
+    the version_number MUST be "1.1.0".


Can we just say that version is the version of the file format?

For example, torflow could add version=1.0.0 and change nothing else, and that would be ok.

teor2345 · 2018-04-21T00:34:20Z

proposals/XX-bw-xxx.txt

+    For documents generated following this specification,
+    the version_number MUST be "1.1.0".
+    Documents generated before this specification do not contain this line,
+    and the version_number is considered to be "1.0.0".


No need to say when the document was generated.
That way, torflow can keep on generating valid documents without this line.

teor2345 · 2018-04-21T00:36:09Z

proposals/XX-bw-xxx.txt

+
+    New implementations SHOULD include this line.
+    Documents generated before this specification do not contain this line,
+    and the software is considered to be "Torflow".


No need to say when the document was generated.
That way, torflow can keep on generating valid documents without this line.

teor2345 · 2018-04-21T00:36:35Z

proposals/XX-bw-xxx.txt

+
+    The version of the software that created the document.
+    New implementations SHOULD include this line.
+    Documents generated before this specification do not contain this line.


No need to say when the document was generated.
That way, torflow can keep on generating valid documents without this line.

teor2345 · 2018-04-21T00:37:54Z

proposals/XX-bw-xxx.txt

+1. Scope and preliminaries
+
+  This document describes the format of Tor's bandwidth measurements
+  document, version 1.1.0 and later.


The document describes version 1.0.0 and later. The current version is 1.1.0.

teor2345 · 2018-04-21T00:38:21Z

proposals/XX-bw-xxx.txt

+    Documents generated before this specification do not contain this line,
+    and the version_number is considered to be "1.0.0".
+    This line is ignored by any Tor version released before this specification.
+    TODO: implement this feature in tor


Please say that this line was added in version 1.1.0.

teor2345 · 2018-04-21T00:38:30Z

proposals/XX-bw-xxx.txt

+    New implementations SHOULD include this line.
+    Documents generated before this specification do not contain this line,
+    and the software is considered to be "Torflow".
+    This line is ignored by any Tor version released before this specification.


Please say that this line was added in version 1.1.0.

teor2345 · 2018-04-21T00:38:34Z

proposals/XX-bw-xxx.txt

+    The version of the software that created the document.
+    New implementations SHOULD include this line.
+    Documents generated before this specification do not contain this line.
+    This line is ignored by any Tor version released before this specification.


Please say that this line was added in version 1.1.0.

teor2345 · 2018-04-29T07:58:51Z

proposals/XX-bw-xxx.txt

+
+    The Unix Epoch time in seconds when the file was created.
+
+  "version="version_number NL


If you like, you can put a space in here:
"version=" version_number NL

teor2345 · 2018-04-29T07:59:42Z

proposals/XX-bw-xxx.txt

+
+    [Zero or one time.]
+
+    The Unix Epoch time in seconds when the file was created.


When the scanner was launched?
Added in 1.1.0.

teor2345 · 2018-04-29T08:00:06Z

proposals/XX-bw-xxx.txt

+
+    [Zero or one time.]
+
+    The Unix Epoch time in seconds when the file was created.


Of the earliest measurement used to create the file?
Added in 1.1.0.

teor2345 · 2018-04-29T08:00:29Z

proposals/XX-bw-xxx.txt

+
+    [Zero or more times.]
+
+    Future releases may include additional key_value header lines.


Future format versions?

teor2345 · 2018-04-29T08:01:40Z

proposals/XX-bw-xxx.txt

+    [One time.]
+
+    The header ends.
+    For documents generated before this specification, the header ends


Please say "version 1.0.0 documents", because torflow will continue to generate documents without newlines after this specification is merged.

teor2345 · 2018-04-29T08:02:12Z

proposals/XX-bw-xxx.txt

+    For documents generated before this specification, the header ends
+    when the first relay measurement line is found conforming to the
+    next section.
+


Added in 1.1.0.

teor2345 · 2018-04-29T08:02:39Z

proposals/XX-bw-xxx.txt

+    next section.
+
+  If a header line does conform to this format, the line SHOULD be ignored by
+  parsers.


This sentence should be up near the key_value paragraph.

proposals/XX-bw-xxx.txt

+software=sbws
+software_version=0.1.0
+scanner_started=1523911756
+earliest_measurement=1523911757


teor2345 · 2018-04-29T08:04:41Z

proposals/XX-bw-xxx.txt

+    Parsers MUST NOT rely on the order of these additional lines.
+    Additional header lines will be accompanied by a minor version increment.
+    Additional header lines MUST NOT use any keywords specified in the
+    relay measurements format.


Scanner implementations may add additional lines by (following a process)...

teor2345 · 2018-04-29T08:23:40Z

proposals/XX-bw-xxx.txt

+    "bw=" bandwidth
+
+  Where bandwidth is the aggregated measured bandwidth of this relay,
+  in kilobytes per second.


Where bandwidth is the aggregated measured bandwidth of this relay.
Torflow scales bandwidths to kilobytes per second. If different
implementations or configurations are used in votes for the same
network, their measurements MAY need further scaling.

And let's put this in Appendix B:

Here is simple scaling method:

Calculate the measured bandwidth per relay by dividing the total
voted bandwidth by the number of relays voted for. In the public tor
network, this is approximately 7500 as of April 2018.

Calculate a quota for the bandwidth authority by multiplying the
per-relay average by the number of relays voted for.

Calculate a scaling factor by dividing the quota by the total
measured bandwidth in this bandwidth authority's upcoming vote.

Multiply each measured bandwidth by the scaling factor.

Now, the total bandwidth in the upcoming vote is approximately
equal to the quota.

If the total bandwidth is zero, all relays should be given equal
weights.

teor2345 · 2018-04-29T08:13:18Z

proposals/XX-bw-xxx.txt

@@ -128,6 +128,8 @@ It consists of:
    Additional header lines will be accompanied by a minor version increment.
    Additional header lines MUST NOT use any keywords specified in the
    relay measurements format.
+    If a header line does conform to this format, the line SHOULD be


Also allow software_version to be any versioning scheme, not just semantic versioning.

Also, expand the bandwidth section. Add additional restrictions that parsers are likely to expect. Reformat the relay section so its intending matches the header section.

from torproject#4 (comment) to torproject#4 (comment)

juga0 · 2018-05-01T15:41:17Z

Work continuing now in https://github.com/juga0/torspec/tree/bandwidth-file-spec branch, as i forgot the first import with the version sent to @Tor-Dev
@teor2345 should i close this PR and open one with that other branch, so it's clear which is the branch there'll be last changes?.

teor2345 · 2018-05-01T21:51:28Z

There are lots of comments on this branch. Let's start again with a new branch.

Initial import, still WIP

68ddcd4

With changes after T. review on tor-dev ml. Unfortunately lost all the granular ci changing line by line :/

teor2345 suggested changes Apr 20, 2018

View reviewed changes

teor2345 reviewed Apr 20, 2018

View reviewed changes

juga0 added 18 commits April 20, 2018 16:40

Center name

a16363a

Change term nonterminals to definitions

3080d80

Uncomment definitions, add timestamp there

67c3290

Explain how header ends

08553d2

Add TODO re. version in tor

98c9ba7

Specify additional args must be key_value

c853b63

Specify seconds for timestamp

af90c43

Specify microseconds for rtt

08e96c2

Specify seconds for timestamp

b7ada67

Explain which arguments torflow uses

65f0b41

Change source for software in example

ebdec2d

Add software and software_version definition

31c4faf

Remove software from definitions

6d8c8e5

Add software in header section

2ca4bc3

Fix timestamp and rtt in example

b0a52ce

Fix indentation

52ef43c

Forgot NL

018703a

Move bandwidth to definitions

91ffb86

teor2345 reviewed Apr 21, 2018

View reviewed changes