As the author of Vote::Count, a standardized format for ballots would be
a big plus. When I've been able to collect sample data, the first thing
I need to do is convert it to my format. Currently Vote::Count has two
formats, a text one for ranked ballots and a json/yaml format for range
ballots. The documentation on my formats is here:
https://metacpan.org/pod/Vote::Count::ReadBallots

I'm not on Reddit, but I think creating a working group of people with
an interest to propose a standard would be  a great idea, and I'm
interested in helping.

A standard format would allow creation of a library of data for which
electowiki would seem to be a natural home.

On 5/27/21 4:02 PM, election-methods-request@lists.electorama.com wrote:

Hi All,

Another source of inspiration could be the Pivot Libre project's Ballot
File Format. Documentation is hosted on GitHub pages.

https://pivot-libre.github.io/bff/

Issues and contributions can be made on GitHub.

https://github.com/pivot-libre/pivot-libre.github.io/issues

I'm interested in staying in the loop on any ranked ballot community
standards efforts. I don't have time to drive the conversation right now.

Whether or not the specifics of BFF are to the community's liking, I
recommend considering using a similar approach with GitHub as a
collaboration and website site hosting platform. I believe several
organizations have used this approach successfully for some fairly complex
standards.

Example:

https://cfconventions.org/index.html
https://github.com/cf-convention

All the best,

Carl

On Thu, May 27, 2021, 3:34 PM John Karr brainbuz@brainbuz.org wrote:

Hi everyone

John, thanks for putting the subject on this thread and for the
pointer to Vote::Count.  Carl and James, thanks for the pointers.
I'll endeavor to keep all of you (and everyone on this mailing list)
in the loop on the progress we're making.  The reddit thread
continued, which I'll repeat the important stuff that I said there.

The name (and file extension) for the format that I'm gravitating
toward is ABIF (".abif"), which stands for "aggregated ballot image
format".  I'm using the term "ballot image" because that seems to be
the term of art for publishing real-world electoral results.  Once
upon a time, "ballot image" meant "a picture of the ballot", but now
just means a crude ASCII representation in a line of text.

I did some processing of the ballot images from San Francisco's 2018
mayoral election, which involved some coding and some manual shell
processing with grep and friends.  My work was ugly the way that all
manual futzing in bash is ugly, but I got a few regexps and some test
data (and some experience) that I'm applying here.  As I was
processing the results, I had wished the results were aggregated in an
easier to process manner.  I would love to finish my processing work
and publish it in a sane format that other programmers can use, which
I'm hoping ABIF can become.

I've been working for WAY TOO MANY years on text formats for
aggregated results.  The format that I used in my Perl script
published in the Perl Journal in 1996 was a noble attempt, and was
better (in many ways) than the revised JSON-based format I published
in 2005 with electowidget.  I think my obsession with JSON was an
unfortunate detour in coming up with a good format.  I've been
studying text formats for structured data for a very long time, ever
since I learned Perl (in 1994 or so) and the work intensified with the
work on RTSP in 1996 through 1998.  We briefly dabbled with making
RTSP a binary protocol using ASN.1, but Dr. Henning Schulzrinne from
Columbia University convinced us (by publishing his "RTSP prime"
draft) that a text-based HTTP knockoff (with MIME headers) was the way
to go.  As we worked on RTSP (and SMIL), we saw the rise of XML, and
the slow steady fading of XML as a data format with the advent of
JSON, and YAML, and TOML, and many other simpler formats.

That's my longwinded way of saying that the test cases that I've
published on the ABIF electowiki page are (I think) a respectable
start for a flexible text format that a wide variety of programmers
can get their heads around:
https://electowiki.org/wiki/ABIF

IThe thing that I love about ABIF (as it's shaping up) is that it
solves several big problems which my 2005 electowidget format didn't.
It goes back to the roots of the 1996 Perl script, back before I "knew
what I was doing", and seems simpler to work with for a reasonably new
programmer (as I was in '96).  My electowidget format REQUIRED
ratings, which I would normalize to rankings as appropriate.  But I
was often trying to express elections that only had ranked ballots
available (e.g. the 2003 Debian election, and the 2009 Burlington
mayoral race).  Having a format that ALLOWS for ratings, but doesn't
require ratings seems appropriate given that IRV/RCV is much more
common in municipal use right now than rating systems.  I love that
the format is very similar to the ad hoc format many have used here on
the EM list for expressing rankings.  I'm reasonably sure that writing
a parser in any language that has reasonable regular expression
support will be easy, and can probably be done with a single-pass
parser.  I haven't really started it yet, but I know how to write a
spec that many programmers can look at, nod their heads, and say
"yeah, I can work with that".  I think having a test suite with
well-specified expected output is going to be a key part of solving
the interoperability problem, and it will be helpful for others to
inspect in a piecemeal fashion rather than feeling obligated to read a
ponderously long specification.

In the next few days, I'll take a look at the implementations that
have been mentioned in this thread.  For example: Pivot's format that
Carl pointed to.  It looks to me that the format that Pivot uses is
very similar to this proposed ABIF format, with the only difference I
see (at first glance) is that this ABIF:

27: A > B > C > D
26: B > A = C > D
24: C > A = D > B
23: D > C > A > B

...would become this in Pivot:
27 * A > B > C > D
26 * B > A = C > D
24 * C > A = D > B
23 * D > C > A > B

Changing colon (":") to asterisk ("*") is an interesting change to
consider.  I suspect that as we all look more closely at Pivot and
other formats, there's going to be other incompatibilities and mindset
differences to hash out.  These all seem like easily solved problems,
because I get the sense that many programmers are hungry for
compatible solutions in this space, and are willing to write
converters to be part of the compatibility party.

At any rate, I think (if others on the mailing list don't mind) that
we should just use this mailing list and electowiki as places to hash
out the format.  If we do this right, it will be easy enough to use
that people on this mailing list (and over on reddit, and many other
places) will keep ABIF compatibility in mind when they write examples
of elections to consider.  Hopefully having more software
compatibility in our ecosystem will make it easier for us to
collaborate on analysis and speed up reform efforts.

Rob

On Thu, May 27, 2021 at 1:34 PM John Karr brainbuz@brainbuz.org wrote:

Hi Rob and all,
I like the simultaneous readability, versatility and robustness of ABIF.
If the multiplier separator is made variant (asterisk / colon / both),
I think the only difference between Pivot and ABIF would be the
significance of whitespace in candidate tokens, something that the
parser could also be liberal in accepting - I see the [] bracketed form
as more readable (and thus default) but no problem in making
the non-bracketed form acceptable.
Also, if leaving out the scores for the unordered variant is allowed,
the format will be able to record approval votes in a very readable
form, which I would support.
If the community eventually arrives at some sort of consensus,
I'll put the ABIF read/write implementation into the backlog of
votelib (which can already read BLT and STV formats) so that
it has an interchange format for ordinal ballots as well.
The implementation would benefit from a formal parsing definition
(BNF or similar); if not, I might create it in the process.
All the best,
Jan

On Fri, May 28, 2021 at 11:32 AM Rob Lanphier roblan@gmail.com wrote:

I too am an aficionado of election data formats, and have been active e.g. on the working groups run by NIST for the Election Assistance Commission, and previous related efforts by OASIS and IEEE years before that.

I appreciate that the goal here is a format that is

• human-readable and easy to enter (e.g. for mailing list discussions), and
• easily parsed (e.g. for software imulations).

The other working groups have been focused on standards for voting system use, and only recently have even handled ranked and rated methods.  E.g. the comprehensive, but verbose and not-very-human-readable CVR standard documented at:

Cast Vote Records Common Data Format Specification Version 1.0 | NIST
https://www.nist.gov/publications/cast-vote-records-common-data-format-specification-version-10

Thankfully we did at least get both ratings and rankings supported there.

At the moment, hopefully early enough in this discussion, I want to push back on the use of the term "ballot image" to mean something other than a graphical representation. The latter is how it is defined in the latest Voluntary Voting System Guidelines 2.0.
https://www.eac.gov/sites/default/files/TestingCertification/Voluntary_Voting_System_Guidelines_Version_2_0.pdf

I know all too well that there was a time (before scanners produced images) that some folks and even standards decided to use the term "ballot image" to mean what we now usually call a "cast vote record". But calling that an "image" is counterintuitivefor the general public, and that usaage has been on the way out for a long time. At the same time, all the recent talk of "ballot images" is about making the graphical representations that all modern voting systems use internally available for quality control and other purposes.

So I'm delighted to see a more convenient format for rated methods, but I'd ask that you not put the word "image" in the name.

Cheers,

Neal McBurnett                http://neal.mcburnett.org/

On Fri, May 28, 2021 at 02:31:25AM -0700, Rob Lanphier wrote:

I agree with Neal that the word "image" should not be part of the ABIF name.

I suggest that ABIF can stand for:

"Aggregated Ballot Information Format"

(And perhaps another appropriate "A" word can be used instead of
"aggregated," if needed.)

I also suggest adding a case number.

A case number allows the ballot data to be processed through separate
vote-counting software while the metadata -- such as precinct number,
political-party affiliations, etc. -- can follow a different path and be
re-joined to produce the published results.

In particular, my vote-counting software focuses on the numbers/counts,
and I use different software (written in my Dashrep programming
language) to process the text info.

The use of a case number also has other benefits.

Otherwise, bravo for making it easier to share and transfer data between
vote-counting programs.

Richard Fobes

On 6/6/2021 1:00 PM, Neal McBurnett wrote:

On 6/6/2021 1:00 PM, Neal McBurnett wrote:

Hi Jan,

Thanks again for reviving this thread.  I agree with you that ABIF has
potential to be simultaneously readable, versatile, and robust, as
well as easy to type for people discussing elections in online forums
(and reasonably easy to embed in an email). We need to be ruthless
about cutting features to keep it simple (more on this in a bit).

There's other stuff to cover in your email, but I'm going to focus on
the asterisk vs colon issue in this email, because I think it's
important to get it right now (before too many parsers and writers get
written).  More below:

On Sun, Jun 6, 2021 at 4:30 AM Jan Šimbera simbera.jan@gmail.com wrote:

My initial instinct is to say "sure!  let's allow many delimiters!",
because that seems easy enough at this stage, and getting
implementations like Pivot on board will help adoption.  But the more
optional representations that we add, the more complicated that
writing parsers for the format is going to be.  Perhaps we should
replace the colon with an asterisk.  I'm not sure.  We should decide
whether we're going to have an optional (non-preferred) delimiter, and
a delimiter that is preferred for software emitting this format.
Small decisions now will have a big impact later.

So let's talk about asterisks.  What you seem to be suggesting is that
ALL of the following test cases should be valid:

A - inlined labels with colon
27: [Doña García Márquez]/5, [Steven B. Jensen]/2, [Sue Ye (蘇業)]/1,
[Adam Muñoz]/0
26: [Doña García Márquez]/3, [Steven B. Jensen]/5, [Sue Ye (蘇業)]/3,
[Adam Muñoz]/1
24: [Doña García Márquez]/2, [Steven B. Jensen]/1, [Sue Ye (蘇業)]/5,
[Adam Muñoz]/2
23: [Doña García Márquez]/1, [Steven B. Jensen]/0, [Sue Ye (蘇業)]/3,
[Adam Muñoz]/5

B - inlined labels with asterisk
27 * [Doña García Márquez]/5, [Steven B. Jensen]/2, [Sue Ye
(蘇業)]/1, [Adam Muñoz]/0
26 * [Doña García Márquez]/3, [Steven B. Jensen]/5, [Sue Ye
(蘇業)]/3, [Adam Muñoz]/1
24 * [Doña García Márquez]/2, [Steven B. Jensen]/1, [Sue Ye
(蘇業)]/5, [Adam Muñoz]/2
23 * [Doña García Márquez]/1, [Steven B. Jensen]/0, [Sue Ye
(蘇業)]/3, [Adam Muñoz]/5

C - labels in header with colon delimiter  [Doña García Márquez]: DGM
[Steven B. Jensen]: SBJ
[Sue Ye (蘇業)]: SY
[Adam Muñoz]: AM

27: DGM/5, SBJ/2, SY/1, AM/0
26: DGM/3, SBJ/5, SY/3, AM/1
24: DGM/2, SBJ/1, SY/5, AM/2
23: DGM/1, SBJ/0, SY/3, AM/5

D - labels in header with asterisk delimiter
[Doña García Márquez]: DGM
[Steven B. Jensen]: SBJ
[Sue Ye (蘇業)]: SY
[Adam Muñoz]: AM

27 * DGM/5, SBJ/2, SY/1, AM/0
26 * DGM/3, SBJ/5, SY/3, AM/1
24 * DGM/2, SBJ/1, SY/5, AM/2
23 * DGM/1, SBJ/0, SY/3, AM/5

(Examples A, C, and D above are equivalent to test cases 4, 5, and 9
on the ABIF wiki page: https://electowiki.org/wiki/ABIF)

All of the examples above (A, B, C, and D) seem easy enough to write
parsing code for.  Still, we need to be opinionated about what format
new ABIF writers should emit, so that we can steer people toward a
single, simple format, and possibly deprecate unused aspects of the
format.  We also want to reduce the complexity of the format now so
that we can add stuff after the format is more widely-used.

You'll note that there's a whitespace difference between examples C
and D.  When using colon, it seems more natural to put the colon
directly after the number.  For asterisk, it seems better to put a
space in to make it clear that it's not intended to be used as a
footnote (like the dagger † and double-dagger ‡ frequently are as
well).  Note that many fonts display asterisks in superscripted
format, which makes it a confusing replacement for the multiplication
symbol ("×") for non-programmers.  Of course programmers look at
asterisk and immediately think of multiplication, but then also start
thinking about the order of operations, and want to see a parenthesis
or two to make it clear that each line isn't being multiplied by the
first candidate.

I'd love to have Pivot on board for ABIF compatibility, so (for now)
I'm publishing a test case on the ABIF page for asterisk being used
instead of colon.  As of right now, I personally prefer colon, but
could be convinced that asterisk is a better choice.  Regardless, we
should decide which one is the preferred delimiter and which one is
the optional delimiter.

I filed this as an issue in the Electorama github issue tracker for ABIF here:
https://github.com/electorama/abif/issues/3

We can continue discussing this on the mailing list, but we can also
discuss it on GitHub.  I'm fine with either for now, since the ABIF
repository on GitHub is new (I made it a few hours ago).

Rob

Hi Neal,

It would be delightful to schedule a phone call or something to talk
about your experience working with NIST/OASIS/IEEE/etc on this.  I've
worked with all of them as well (a little bit), but my focus (when I
was doing standards stuff) was IETF and W3C, and it was unrelated to
voting systems.  I was paying attention to this stuff when W3C
splintered off of IETF, and it's been interesting watching WHATWG's
on-again/off-again relationship with W3C.  But we should take that
discussion off of this list.

There's a lot to talk about in your email, but I'm going to zero in on
one particular item for this email:

On Sun, Jun 6, 2021 at 1:01 PM Neal McBurnett neal@bcn.boulder.co.us wrote:

As much as I enjoy naming discussions (ALMOST more than I enjoy eating
glass), you're right.  Now is the time to change the name if it's
going to get changed.  As the old joke goes, there are only two hard
problems in computer science: cache invalidation, naming things, and
off-by-one errors.  We'll focus on the second one for now. :-)

Speaking of "two hard problems", I've given this the honorary "#2"
spot in the issue tracker:
https://github.com/electorama/abif/issues/2

Something tells me we're going to have more than two hard problems
with ABIF.  I still want to call it ABIF even if we decide that "I"
doesn't stand for "Image".  We have other alternatives (more on that
as a bit).

The reason for sticking with "ballot image" in the name:  it seems
likely that some jurisdictions (like San Francisco) may have codified
the term "ballot image" to refer to a single line of ASCII text
representing a ballot, based on this:
https://sfelections.sfgov.org/june-5-2018-election-results-detailed-reports

...and this:
https://www.sfelections.org/results/20180605/data/BallotImageRCVhelp.pdf

It seems as though it's a term of art that has been used in many
places, even if it doesn't make sense.  Given that San Francisco was
arguably where the term "Ranked Choice Voting" was invented (and they
got FairVote to change from "Instant Runoff Voting" to that name), I'm
not sure I want to try to fight the powers that be in SF on a naming
dispute.

In fairness to your point of view, it would seem that the "Voluntary
Voting System Guidelines" that you cited (published by Election
Assistance Commission - EAC) seems to draw a distinction between
"ballot image" and "ballot record":
https://www.eac.gov/sites/default/files/TestingCertification/Voluntary_Voting_System_Guidelines_Version_2_0.pdf

That's a pretty compelling reason to stop using "ballot image" for
anything that isn't a scan or photograph of some variety.

This could become a BRAF (Ballot Record Aggregation Format).  I like
that name too, but I'll still call it ABIF unless there's a flurry of
BRAF support.  I like having "Aggregation" in the name to make it
clear that this format is for (optionally) expressing the stated
preferences of many voters (hundreds, thousands, more) in a single,
readable line of ASCII text.

It seems like we can make "ABIF" work. Renaming ABIF at this point is
going to create some busywork for me; not nearly as much as it would
if we wait, but I've already have many little pockets of things with
"ABIF" on it (e.g. https://github.com/electorama/abif ).  Let's not
rename this thing a lot. I'm okay with changing what the "I" stands
for (if we really must)  "Ballot Information", "Ballot Inventory", and
"Ballot Igloo" all work for me (well, maybe only two of those three,
but you'll need to guess which one)  :-D

Assuming you're signed up on GitHub, let's continue the conversation there:
https://github.com/electorama/abif/issues/2

We can also use the mailing list, but I suspect that other mailing
list members will appreciate this list not becoming the ABIF naming
mailing list (or the BRAF mailing list, or whatever we call it).

Rob