Handle votes with joint responsible committees #1122
Conversation
tillprochaska
force-pushed
the
multiple-responsible-committees
branch
2 times, most recently
from
6f66b36 to
57f9698
Compare
There was a problem hiding this comment.
More of a conceptual objection: I think extracting the responsible committees from the votes.csv severely limits the usefulness of that file and makes finding stuff on specific topics (or maybe better: limiting the list of votes to sensible candidates) much harder.
I know that one can just join the two tables, but I am thinking about a quick Excel-thing here.
There are multiple ways to handle this imo:
- One can say does not matter and just do it anyways, and rethink this once negative feedback comes in. I think relying on this is not a good solution, in this case.
- I would need to spend some more time on reading the procedural rules, but as long as there is a hard cap on the number of responsible committees, we could just include several columns in the votes.csv, most of them empty.
- I even would be okay with a solution that just has one columns and string-concatenates them ("LIBE, IMCO", as an example).
Side-note: As its cheap, should we also collect the Committees for opinion? Probably should not even be in this PR and here I would also be totally fine with extracting that in a separate table in any case, possibly also duplicating information in the case we decide to leave the responsible committees in the votes.csv. We have duplicated information in the export anyways. Just wanted to raise the matter here as it seemed fitting.
tillprochaska
force-pushed
the
multiple-responsible-committees
branch
5 times, most recently
from
faa2cf2 to
7eae4cf
Compare
tillprochaska
force-pushed
the
multiple-responsible-committees
branch
from
7eae4cf to
e1faf83
Compare
| with self.committees.open() as committees: | ||
| exp = func.json_each(Vote.responsible_committees).table_valued("value") | ||
| query = ( | ||
| select(func.distinct(exp.c.value)).select_from(Vote, exp).order_by(exp.c.value) | ||
| ) | ||
| committee_codes = Session.execute(query).scalars() | ||
|
|
||
| for committee_code in committee_codes: | ||
| committee = Committee[committee_code] if True else None | ||
| committees.write_row( | ||
| { | ||
| "code": committee.code, | ||
| "label": committee.label, | ||
| "abbreviation": committee.abbreviation, | ||
| } | ||
| ) |
There was a problem hiding this comment.
Just for my understanding - this means that the committee table will not be guaranteed to contain all committes, but all committees that were responsible for a vote that we collected, correct?
There was a problem hiding this comment.
Yes, exactly. The list of committees we obtain from the Publications Office contains every committee that has ever exists (some of them which have existed for only a very short amount of time decades ago). This is similar to what we do for e.g. EuroVoc concepts as well, we only export what’s actually relevant to the dataset.
A procedure can have multiple responsible committees (see Rules of Procedure and an example procedure).
responsible_committee_codein thevotes.csvtable. Instead, there are now separate tablescommittees.csvandresponsible_committee_votes.csv.After deployment, we need to re-scrape the procedure files and then re-aggregate the votes.
Fixes #1121