Assigning Posts to Annotators

[tarunima]

From all data that is collected from twitter, a sample of data is used for annotations. This data will be stored in a SQL database and made available via a REST API. Each post needs to be allocated to possibly more than one annotator.

We need to write a script to allocate posts to annotators in a way that certain conditions are met.

Method Signature

The method signature should be of the form allocate(posts, annotator). The shape of posts and annotators is as follows :

posts : [
  {
    tweet,
    hashtags,
    timestamp,
    language
    annotations: [
      {
        id,
        status : pending | completed
      }
    ]
 }
]

annotators: [
  id,
  language,
  max_post,
  overlap : # yet to be defined
]

Distribution Criteria

k% of the posts need to be annotated by 3 annotators. The rest of the posts (100-k)% will be annotated by one annotator.

Doubts to be clarified

Is it necessary that every annotator's annotations are included in agreement estimation. So at least m% of an annotator's posts must have been annotated by at least one other annotator?

[dennyabrain]

This might be useful.

This a screenshot of the table that keeps track of which posts are assigned to which user and what the annotation's status is. The output of allocate() should be an array with each array item being {userId, postId}.

[mlkorra]

can't we fetch data using mongo-db instead of SQL since we are already using that for storing the posts meta data?

[dennyabrain]

Unlike the data in mongo which is mostly written to and read, I foresee the need to run a lot of queries involving grouping, filtering and updates for the annotator UI, so i chose SQL.
Regardless you wouldn't have to worry about the SQL integration yet. I might provide you a REST API to work with. for now, lets just focus on the annotate function and you can assume that the function receives data in the formats described above.

[mlkorra]

I see. That makes sense

[mlkorra]

How Is the parameter K assigned for now,is it given by the user as input?

[dennyabrain]

oh, good catch. I think allocate's method signature should include a variable called config. k should be in config. maybe config.net_overlap or something. open to other naming suggestions.