-
Notifications
You must be signed in to change notification settings - Fork 23
Observer Manipulation
jojenki edited this page Jun 4, 2012
·
33 revisions
This feature is still experimental. Its definition may change at any time, without notice, and the data collected by it may be deleted on the development machines at any time.
An observer is a set of streams that define sets of data collected. This is analogous to our campaigns, where a campaign defines an overall set of surveys for a user.
A stream is a definition of a set of data to be collected. This is analogous to our surveys, where a survey defines a specific set of questions to ask a user.
Presently, all of our observers are defined in XML and the schema of each component is an Avro definition. The XML should have a root tag that is "observer" and is defined as follows:
| Key | Description | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | A unique ID for the observer. This should follow the Java naming conventions with periods as dividers between parts of the name, e.g. "org.ohmage.my.probe". It must begin with an alphanumeric character and may contain any combination of alphanumeric characters including periods to use as a separator; however, two periods may not appear together. The maximum length is 255 characters. | ||||||||||||||
| version | A number describing this observer's version. This value must be increased any time any part of the observer's definition is changed. | ||||||||||||||
| name | A user-friendly name for this probe. | ||||||||||||||
| description | A user-friendly description describing the overall purpose of this probe. Each stream will have its own description, which is the more applicable place to describe exactly what type of data is being collected. | ||||||||||||||
| versionString | A user-friendly string representing the version of the probe. | ||||||||||||||
| stream | Any number of these tags may exist as long as there is at least one. These are the individual streams of data being collected. They are defined as follows:
|
<?xml version="1.0" encoding="UTF-8"?>
<observer>
<id>edu.ucla.cens.Mobility</id>
<version>2012050700</version>
<name>Mobility</name>
<description>The Mobility probe collects the user's current movement type (still, walking, running, etc.), which we call "mode", and may also collect the phone's movements.</description>
<versionString>3.0</versionString>
<stream>
<id>regular</id>
<version>2012050700</version>
<name>Mobility - Regular</name>
<description>This records only the user's mode.</description>
<metadata>
<timestamp />
<location />
</metadata>
<schema>
{
"type":"record",
"namespace":"edu.ucla.cens.MobilityRegular",
"name":"regular",
"doc":"Only contains the user's mode.",
"fields":[
{
"name":"mode",
"doc":"The user's mode.",
"type":"string"
}
]
}
</schema>
</stream>
<stream>
<id>extended</id>
<version>2012050700</version>
<name>Mobility - Extended</name>
<description>This records the user's mode as well as accelerometer, WiFi, and GPS data.</description>
<metadata>
<timestamp />
<location />
</metadata>
<schema>
{
"type":"record",
"namespace":"edu.ucla.cens.MobilityExtended",
"name":"extended",
"doc":"Only contains the user's mode.",
"fields":[
{
"name":"mode",
"doc":"The user's mode.",
"type":"string"
},
{
"name":"speed",
"doc":"The user's speed in the last minute.",
"type":"double"
},
{
"name":"accel_data",
"doc":"An array of the accelerometer readings over the last minute.",
"type":{
"type":"array",
"items":{
"type":"record",
"name":"accel_point",
"doc":"A single accelerometer reading.",
"fields":[
{
"name":"x",
"doc":"The x-component of the accelerometer reading.",
"type":"double"
},
{
"name":"y",
"doc":"The y-component of the accelerometer reading.",
"type":"double"
},
{
"name":"z",
"doc":"The z-component of the accelerometer reading.",
"type":"double"
}
]
}
}
},
{
"name":"wifi_data",
"doc":"A WiFi reading from the last minute.",
"type":{
"type":"record",
"name":"wifi_point",
"doc":"A WiFi reading from the last minute.",
"fields":[
{
"name":"time",
"doc":"The time this data was recorded.",
"type":"long"
},
{
"name":"timezone",
"doc":"The time zone of the device when this data was recorded.",
"type":"string"
},
{
"name":"scan",
"doc":"The scan of WiFi information.",
"type":{
"type":"array",
"items":{
"type":"record",
"name":"access_point",
"doc":"A single access point's information.",
"fields":[
{
"name":"ssid",
"doc":"The access point's SSID.",
"type":"string"
},
{
"name":"strength",
"doc":"The strength of the signal from the access point.",
"type":"double"
}
]
}
}
}
]
}
}
]
}
</schema>
</stream>
</observer>
Allows a user to create a new observer.
observer/create
Anyone is allowed to create observers.
- (r) user = The user's username.
- (r) password = The user's password.
OR
- (r) auth_token = The user's authentication token.
- (r) observer_definition = The observer's definition as defined above.
POST /app/observer/create HTTP/1.1
Host: dev.ohmage.org
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7
Content-Length: byte-length-of-content
Content-Type: application/x-www-form-urlencoded
auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e
&observer_definition=<XML Content>
curl -v -F "auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e" -F "observer_definition=@/myObserver.xml;type=text/xml" http://localhost:8080/app/observer/create
{
"result" : "success"
}
See the error page for a description of error codes and their associated descriptions.
Uploads a set of data points for a user. All of the points must belong to the same observer with the same version, but may belong to any of the streams defined in that observer-version combination.
stream/upload
Anyone is allowed to upload their own data to an existing observer.
- (r) user = The user's username.
- (r) password = The user's password.
OR
- (r) auth_token = The user's authentication token.
- (r) observer_id = The unique identifier for this observer.
- (r) observer_version = The version of this observer to which this data applied.
- (r) data = The data to be uploaded. This have the format described below.
The data must be a JSON array of JSON objects. Each object must follow this definition:
| Key | Description | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| stream_id | The unique identifier for the stream to which this data applies. | ||||||||||||||||||||||||
| stream_version | The version of this stream to which this data applies. Because the observer's ID and version and the stream's ID were already supplied, this value is redundant. The reasoning behind it is a second layer of validation. When an observer's version changes, it is not a requirement that all, or any, streams change. Therefore, this stream ID-version may have been valid in previous versions of the observer, and, therefore, so would the data associated with it. Conversely, if this stream did change and the change was only a change in the name of a field in this stream's schema but not its type, then data that was valid with the previous schema would also be syntactically valid for this schema. However, it may not be semantically valid; for example, if a field was changed from "first_name" to "last_name" but the type remained a string, then old data from an outdated client that is returning the first name would be saved as the last name, which, while syntactically correct, is not semantically correct. Therefore, this reiteration of the version gives us a stronger guarantee that this data does apply to this stream's ID-version combination. | ||||||||||||||||||||||||
| metadata | This should be a JSON object containing the metadata for this point. This field is optional, even if the definition states that records may contain metadata. The format of this object is:
|
||||||||||||||||||||||||
| data | The data according which must match this stream's definition. Additional fields may be provided, but the defined fields must be present. The additional fields will be saved. |
POST /app/stream/upload HTTP/1.1
Host: dev.ohmage.org
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7
Content-Length: byte-length-of-content
Content-Type: application/x-www-form-urlencoded
auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e
&observer_id=org.ohmage.myObserver
&observer_version=123456789
&data=<The data>
curl -v -F "auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e" -F "observer_id=org.ohmage.myObserver" -F "observer_version=123456789" -F "data=@/my.data;type=text/xml" http://localhost:8080/app/stream/upload
{
"result" : "success"
}
See the error page for a description of error codes and their associated descriptions.
Reads the contents of a stream. For now, we only allow a single stream's data to be read at a time. The reason for this is that all of the parameters are specific to each stream read, so allowing multiple streams to be read simultaneously would require that the stream ID, stream version, and column list would need to be duplicated for each stream.
stream/read
Anyone is allowed to read their own data. For now, we don't have a way to allow users to read data about other users.
- (r) user = The user's username.
- (r) password = The user's password.
OR
- (r) auth_token = The user's authentication token.
- (r) observer_id = The unique identifier for this observer.
- (o) observer_version = The version of the observer. Because a stream may not change between observer version changes, then querying a stream with a specific version and not providing this parameter would allow you to read across observer versions where the stream version didn't change.
- (r) stream_id = The unique identifier for the steam.
- (r) stream_version = The version of the stream.
- (o) start_date = Limits the results to only those after or on this date. This must be an ISO8601 date-time of the form "yyyy-MM-dd'T'HH:mm:ss.SSSZZ".
- (o) end_date = Limits the results to only those before or on this date. This must be an ISO8601 date-time of the form "yyyy-MM-dd'T'HH:mm:ss.SSSZZ".
- (o) column_list = Limits the results to only those whose columns are in this list. The format of the list is a comma-separated set of strings. Each string denotes the column to return based on the following schema, :::.... For a few record of the form {"a":"b","one":{"sub1":"first", "sub2":"second", "sub3":[{"sub-sub1":1, "sub-sub2":2}, {"sub-sub1":10, "sub-sub3":30}]}}, the column_list a, one:sub1, one:sub3:sub-sub2 would return {"a":"b", "one":{"sub1":"first", "sub3":[{"sub-sub2":2}, {}]}}. The columns do not need to be defined in the schema, so additional data that was added may be queried, but records without that data will be empty but present, i.e. {}.
- (o) num_to_skip = The number of records to skip. If this is negative, it will be set to 0. It is valid to skip more records than exist, but the number of records returned will be 0.
- (o) num_to_return = The number of records to return. If this is negative or greater than the maximum, it will be reset to the maximum. The current maximum is 2000.
POST /app/stream/read HTTP/1.1
Host: dev.ohmage.org
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7
Content-Length: byte-length-of-content
Content-Type: application/x-www-form-urlencoded
auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e
&observer_id=org.ohmage.myObserver
&stream_id=myStream
&stream_version=123456789
&column_list=some_int,some_record:sub_item,some_array:sub_item:sub_sub_item
curl -v -F "auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e" -F "observer_id=org.ohmage.myObserver" -F "stream_id=myStream" -F "stream_version=123456789" -F "column_list=some_int,some_record:sub_item,some_array:sub_item:sub_sub_item" http://localhost:8080/app/stream/read
{
"result" : "success"
"metadata":{
"count":<A number representing the number of results.>,
"prev":"<The URL for the previous set of results.>",
"next":"<The URL for the next set of results.>"
},
"data":[
{
"metadata":{ // If no metadata exists, this key will not exist.
"timestamp":"<The ISO8601 timestamp.>" // If no timestamp exists, this key will not exist.
"location":{ // If no location exists, this key will not exist.
"latitude":12.345
"longitude":67.890
"accuracy":15
"provider":"Magic"
}
},
"data":{} // The data based on the given columns and the data collected for this point.
},
...
]
}
See the error page for a description of error codes and their associated descriptions.